All Your Code Are Belong to Me

Publishing of this post was postponed until the report vulnerability described here had been corrected. The core risk of software supply chain issues is still a widespread issue.

tldr; don’t depend on un-pinned external projects controlled by someone who doesn’t work at your company. Especially as a core part of your payment processing infrastructure.

Fun security-related thing that I thought I’d share, just so that people don’t forget the risk.

I was at a company years ago that, as an e-commerce platform, did a lot of payment processing. At the time, we found an annoying bug in an open-source library we were using, and decided to fork the repository, patch the issue, and use our own fork going forward. At the time I made this fork under my own github rather than the company github (whoops), and it was referenced as a python git wheel against the branch. This was a bad practice, but it worked at the time, we were a small start-up, and it did not get fixed.

Yes, this is a bad practice, but we all know it happens in small companies, and we shouldn’t pretend it doesn’t. If we do pretend, then we are just ignoring the lessons that can be learned.

Fast-forward to last 2020. I was going through repos, and saw one that I didn’t recognize. I was about to delete it when I remembered what it was. Before deleting it, I reached out to a friend to make sure that they had updated the dependency since my time there, and that my fork wasn’t still in use.

They hadn’t. It was 100% still in use.

Now I’m a nice person. I obviously was not going to do anything malicious here. But really, the best case scenario of me changing something here would have been that their usage stopped working, or their build breaks.

The malicious scenario was that all of the customer monies would now belong to me.

It would have been trivial to push a new version of this package that would have been included into this companies build, and redirect arbitrary payments away from their intended destination. It would have been trivial to monkey-patch any code running within the application to exfiltrate data.

Well your first thought might be “if you made a malicious change to an open source package, someone would have immediately noticed.” Yes, for an open-source product that would be the case. But for a fork of a public project which is used by only a single company? Unlikely. Either way, you should not be depending on detecting changes in dependencies after-the-fact, but rather ensuring that the versions and signatures of all of your dependencies are pinned.

So, ya, software supply chain matters. But what can we do about it?

1. Do not use un-pinned dependencies

I understand that with modern tools this is often very difficult, due to the deeply nested dependency chains that have developed as a result of modern package managers (NPM, Pip, etc). I do not have any full solutions here, but there are plenty of other resources which attempt to present best practices here.

2. Vendor any dependencies which cannot be pinned

Related to the above, in a situation where it is impractical to pin all dependencies in order to provide a reproducible build you should be vendoring your dependencies. Obviously this can make updating dependencies a very tedious process, and a balance will have to be made here taking into account your specific development velocity and security needs.

3. Do not use forks which are not controlled internally

Forks of public projects are largely under much less scrutiny than the “official” version. If you need to depend on a fork, it should live in the corporate version control system (in Github, under the organization’s account). Forks under personal accounts should not be used.

There may be little difference initially, but as soon as that employee leaves, the organization effectively loses control of the fork, and immediately begin using a dependency which is controlled by a potentially-malicious external individual. This is always bad.

Keeping all company-written code within the company-controlled version control system ensures that employees do not maintain the ability to push arbitrary code into production after a departure.

Hopefully this was helpful for you. If you have any questions, don’t hesitate to shoot me an email, or follow me on twitter @nrmitchi.