At PatchFox we spend a lot of time applying patches to vulnerable code. A vulnerability is disclosed to the National Vulnerability Database, the vendor releases a patch that fixes the issue, and we apply the patch. In most cases, for most reputable vendors, the patch is released soon after the defect is disclosed to the NVD. Most but not all. Every once in a while the vendor refuses to issue a patch. This is about my encounter with one such defect, the “15-year-old Python tarfile flaw [that] lurks in over 350,000 code projects”.

The issue was first publicly identified in 2007 by Jan Matejek on the python mailing list. It demonstrated how, despite the underlying tar utility being patched years prior, the module that provides a python interface to it was still vulnerable. From this CVE-2007-4559 was born (if you want to see for yourself how the exploit works, check out our demo here).

To put how long ago this was in perspective, in 2007 George W. Bush was President of the United States. The first iPhone was released. Caitlyn Jenner still went by Bruce. When all is said and done it would take the python folks sixteen years to produce a patch. In that time, according to Trellix, a mammoth amount of vulnerable code has proliferated. While it would be easy to take from this the lesson that the python folks screwed up I don’t think that’s fair. This wasn’t negligence. This was passion. This was conviction. This was love.

In his personal statement regarding all of this, Lars Gustäbel, maintainer of the vulnerable python library until 2019, wrote:

“The tarfile module does exactly what it is supposed to do when it is executing the instructions stored in the tar archive. The alleged security vulnerability in tarfile is not the result of a mistake or negligence. tarfile does what it does by design. I wrote some more on that here in 2014.”

“I believe it is necessary and unavoidable that programmers that use other people’s code in their projects have to take responsibility to some degree. They should make themselves familiar with what the code does, how it works and what the possible risks are. They should take the trouble to read at least some amount of the documentation.”

First and foremost – I would be remiss if I didn’t point out the first statement isn’t entirely true. Tar has for years implemented default behavior that thwarts the symlink attack described in CVE-2007-4559. When directly invoked at the terminal, you have to override that default behavior with the “-P” option in order for you to be susceptible. When activating tar through the python tar library this default no longer existed. That being said – that’s a fact that doesn’t change what’s important. What’s important is that Lars had perspective he was invested in and was willing to fight for.

The simple statement is that we, as engineers, everyday, use other people’s code in combination with our own. Everything we build uses some library, some interface, some tool written by other people. It’s a paradigm that requires some measure, and balance, of trust but verify. Lars placed emphasis on the verify. His view was that tar is a powerful tool that needs to be wielded responsibly and that python, as the go-between, should not be in the business of getting between the engineer and the tool. In principle that is a fair-minded, widely-shared, value among the engineering community.

Where things get challenging is how to live that value. It’s simply not possible for an industrial engineer to read every release note, every manual, every stack overflow thread, etc, to deeply and fully grok every nuance of every element of their stack. There are too many things and too few hours in the day. There’s also a pesky need for sleep and the occasional beer. The simple statement is sometimes “trust” is leaned on more heavily than “verify” when product needs to be shipped.

I stumbled into all of this while working for a large company with a lot of old python. I was leading a team tasked with replacing a legacy python monolith whose job was to inspect tarballs from third-party developers. In creating the new system we looked at how the legacy one did things and the code appeared robust. It performed several sanitation checks against the artifact before unpacking it and there was nothing at the time in the python docs indicating the approach would be problematic. We ported the code over and called it a day… or would have, had it not been for my colleague Aaron James Eason.

Aaron was part of a community that makes Capture the Flag (CtF) exercises. He was the one who implemented the tar validation logic and used it to make a CtF challenge involving tarslips. He needed a guinea pig to give his creation a try and I volunteered. We went a few rounds of my defeating the box and him tweaking it when Aaron realized it was easier than we thought to smuggle malicious tarball nasty past the usual python countermeasures and I realized the legacy system that – at that point was still the production system and still processing hundreds of untrusted tarballs a day – was vulnerable.

In short order we refactored all the things so we were no longer vulnerable. It wasn’t easy as tar is a wild and arcane beast when you get down to it. It’s an archival tool and as such it picks up on things on the file system most of you wouldn’t consider “files”. We were lucky in that we discovered the problem before anyone did anything weird. In a world where it’s impossible to know everything about everything, where diligence is a marathon of doing your best with what you have, it’s not practical to exercise due and complete diligence in every circumstance. Lars, I think, would agree. It’s a shared responsibility. The disconnect that caused CVE-2007-4559 to remain un-patched for over a decade-and-a-half was intransigence in point of view as to who bore what share of that burden. We got caught in the middle.

It is probably true that Lars’ strongly held view that the users of his library bore more responsibility for avoiding common pitfalls than he did contributed to the degree to which CVE-2007-4559 metastasized. That being said, while we didn’t do it on purpose, our success in thwarting disaster was the result of doing exactly what Lars trusted we would do – validate our trusted assumptions with diligence in pursuit of fact. It wasn’t until we actually tried defeating what we thought was secure that we understood it wasn’t – and why. In other words, sometimes RTFM is not enough. Taking a beat to try and break what you built is an effective tool that can be used to validate the trust you have in people like Lars and the efficacy of what they made when called by your code.

It’s hard and it’s dry and it can be cruel sometimes

but it might just save your hide

that’s the power of p0wn.

(ps – CVE-2007-4559 won’t show up on most of your OSS scanners. If you’ve got python and tar you’re probably vulnerable. Send us a message at ask@patchfox.io and we’d be delighted to help!)