The Internet Was Weeks Away From Disaster and Nobody Knew
In late March 2024, the world came short distance of the most devastating cyberattack in history. Not because of a zero-day exploit in a Fortune 500 company’s software, not because of a brute-force assault on government servers, but because a single, quietly maintained open-source compression utility called XZ Utils had been turned into an entry key for nearly every Linux server on the internet. As Veritasium detailed in their documentary-style breakdown, The Internet Was Weeks Away From Disaster and No One Knew, the story reads less like a security advisory and more like a slow-burning espionage thriller — one that was only foiled by a curious engineer who noticed a half-second delay.
The Invisible Backbone You’ve Never Heard Of
XZ Utils is one of those pieces of software that almost nobody outside the Linux world thinks or talks about. It handles LZMA (Lempel–Ziv–Markov chain algorithm) compression and decompression, an ordinary but essential function integrated into almost every Linux distribution. If you’ve ever downloaded a software package, extracted a tarball, or booted a Linux server, XZ Utils was probably involved somewhere. It is, in every meaningful sense, critical infrastructure.
And for years, it was maintained almost entirely by one person: Lasse Collin, an unpaid volunteer. His official response to the incident confirmed the scope of the compromise and detailed the steps taken to regain control of the project.
This is the uncomfortable truth at the heart of the modern internet that Veritasium’s video drives home. Trillion-dollar corporations such as banks, cloud providers, and governments run their operations on software maintained by individuals who do it out of passion, not paychecks. As the famous xkcd comic about open-source dependencies illustrates, the entire digital economy sometimes rests on the shoulders of “some random person in Nebraska.” In this case, that person was a Finnish developer who had been maintaining XZ Utils since 2009.
Enter Jia Tan: The Long Con
In 2021, a GitHub account appeared under the username JiaT75. The person behind it was using the name “Jia Tan” and began making small, helpful contributions to open-source projects. According to Russ Cox’s detailed timeline, their first known commit was to the libarchive project, where they submitted a seemingly innocent patch. In retrospect, the change actually replaced a safer function with a less secure variant. Nobody noticed.
Some of these accounts attempted to present themselves as being in Beijing, China (UTC + 8), but analysis of commit timestamps revealed they were actually operating in a UTC + 2 timezone, a region covering parts of Eastern Europe, North Africa, and the Middle East. This added another layer of deception, showing the care in masking their true location.
Countries in UTC + 2 (Standard Time / Central Africa / Eastern European Time)
- Europe (Eastern European Time):
- Finland, Estonia, Latvia, Lithuania, Greece, Bulgaria, Romania
- Parts of Ukraine and Moldova
- Africa (Central Africa Time):
- Egypt, Libya, South Africa, Sudan, Malawi, Zambia, Zimbabwe, Mozambique, Botswana, Namibia
- Middle East:
- Some parts of Syria, Jordan, and Lebanon (when not observing daylight savings)
Note: Many of these countries switch to daylight saving time in summer, moving to UTC + 3, but the UTC + 2 designation is a good baseline for general attribution.
By 2022, Jia Tan had turned their attention to XZ Utils. They submitted patches, participated in mailing list discussions, and began building a reputation as a reliable contributor in the community. Almost simultaneously, previously unseen accounts such as “Jigar Kumar,” “Dennis Ens,” and others began pressuring Lasse Collin on the mailing list, complaining that updates were too slow and that the project needed a new co-maintainer. Kaspersky’s Securelist analysis describes this as a coordinated social engineering campaign: create artificial urgency, manufacture community pressure, and present your operative as the obvious solution.
Collin, dealing with burnout and mental health challenges he had openly discussed, eventually granted Jia Tan co-maintainer status. Over the following months, Tan methodically expanded their control gaining commit access, release manager rights, and even replacing Collin’s contact information on Google’s oss-fuzz security testing platform. Critically, Tan also disabled ifunc support in oss-fuzz testing for XZ Utils, a move that would later prevent automated tools from detecting the malicious changes to come.
This wasn’t just a hack. It was a years-long infiltration.
The Backdoor
In February 2024, Jia Tan carefully merged backdoor code into XZ Utils versions 5.6.0 and, later, 5.6.1. As Akamai’s technical breakdown details, the attack was astonishingly sophisticated. The malicious payload wasn’t even visible in the project’s public Git repository as it was embedded only in the source code tarballs used for official releases, hidden inside what appeared to be binary test files.
When compiled under the right conditions, the backdoor hijacked the build process using a technique involving IFUNCs (indirect functions) to modify how the resulting library behaved. The target was OpenSSH’s sshd daemon. On many major Linux distributions, including Debian, Ubuntu, and Fedora, sshd is patched to link against systemd, which in turn links against liblzma (part of XZ Utils). This chain of dependencies meant that compromising XZ Utils effectively compromised SSH on millions of servers.
The backdoor allowed anyone holding a specific Ed448 cryptographic key to execute arbitrary commands on a target machine, without authentication, simply by connecting to its SSH port. The vulnerability was assigned CVE-2024-3094 with a perfect CVSS score of 10.0 — the most severe rating possible.
The Race to Ship
With the malicious code in place, the pressure campaign shifted to distribution. Jia Tan and their suspected sock puppet accounts began lobbying Linux distribution maintainers to adopt the compromised versions. “Hans Jansen”, yet another fictitious identity linked to the operation by Securelist that filed a Debian bug report urging an update to version 5.6.1. Meanwhile, Tan pushed for inclusion in Fedora 40’s pre-release builds and attempted to get the code into Ubuntu 24.04 LTS.
According to Datadog Security Labs, the compromised version made it into several bleeding-edge and pre-release distributions, including Fedora Linux 40 beta, Fedora Rawhide, Debian unstable, and Arch Linux. Red Hat issued an urgent security alert telling users to “immediately stop using Fedora 40 or Fedora Rawhide” until they could downgrade. Canonical delayed the Ubuntu 24.04 LTS beta by a full week and opted for a complete binary rebuild of every package in the distribution as a precaution.
Had the backdoor gone undetected for just a few more weeks, it would have shipped in stable releases used by enterprises, cloud providers, and governments across the globe.
A 500-Millisecond Miracle
The person who stopped all of this was Andres Freund, a Microsoft engineer and PostgreSQL developer based in San Francisco. As Elastic Security Labs documented in their aptly titled report 500ms to Midnight, Freund noticed that SSH logins on his Debian development system were consuming an unusual amount of CPU, and connections were taking roughly 500 milliseconds longer than expected. Half a second. Most people would have shrugged it off.
Freund didn’t. He pulled the thread, traced the performance regression to the liblzma library, and methodically reverse-engineered what he found. On March 29, 2024, he posted his findings to the oss-security mailing list, calmly explaining that one of the most widely used compression utilities in the Linux ecosystem had been deliberately backdoored for remote code execution. (Freund later clarified on X/Twitter that his investigation actually started after noticing abnormal CPU usage from failed SSH login attempts, and the 500ms latency was a secondary observation.)
The cybersecurity world erupted. GitHub suspended the XZ Utils repository. CISA issued an emergency advisory. Distribution maintainers scrambled to revert to safe versions.
What It Means
The XZ Utils incident forced an industry-wide reckoning. The attack is widely believed to be the work of a state-sponsored actor. Security researcher Dave Aitel has suggested it fits the pattern of APT29, tied to Russia’s SVR. The operational security, multi-year timeline, and use of multiple fictitious identities all point to the resources and patience of a nation-state intelligence operation. SentinelOne’s analysis further revealed that the attacker was planning to inject additional backdoors, including code to disable Linux’s LandLock sandboxing which suggests the XZ compromise was just the beginning.
But the deeper lesson isn’t about espionage. It’s about the fragility of the open-source ecosystem. As Veritasium’s video emphasises, we’ve built the modern internet on software maintained by volunteers who are often overworked, under-appreciated, and one bad day away from burning out, which is exactly the vulnerability that Jia Tan exploited. Lasse Collin wasn’t negligent. He was a solo maintainer doing his best, and the attackers weaponised his humanity against him.
The Open Source Security Foundation and OpenJS Foundation issued joint warnings that the XZ backdoor may not be an isolated event, reporting that similar social engineering attempts had targeted JavaScript projects. Bruce Schneier, writing on his security blog, called it an “incredibly complex backdoor” that involved social engineering a lone unpaid engineer over multiple years. And as recently as August 2025, Binarly researchers found Debian Docker images on Docker Hub still containing the compromised XZ Utils code.
“We were incredibly lucky this time,” Andres Freund wrote on Mastodon after the disclosure. “But relying on luck in the future is a bad strategy.”
He’s right. Next time, the curious engineer might not notice the extra half-second. And next time, the backdoor might already be in production.
Sources & Further Reading
- Veritasium — The Internet Was Weeks Away From Disaster and No One Knew (YouTube)
- Wikipedia — XZ Utils backdoor
- Lasse Collin / Tukaani Project — Official XZ Backdoor Statement
- Russ Cox (research!rsc) — Timeline of the xz open source attack
- Sam James (@thesamesam) — xz-utils backdoor situation (CVE-2024-3094) (GitHub Gist)
- Akamai Security Research — XZ Utils Backdoor: Everything You Need to Know
- Datadog Security Labs — The XZ Utils backdoor (CVE-2024-3094)
- Elastic Security Labs — 500ms to Midnight
- Kaspersky / Securelist — Social engineering aspect of the XZ incident
- SentinelOne — XZ Utils Backdoor: Threat Actor Planned Further Vulnerabilities
- Qualys — CVE-2024-3094: XZ Utils SSHd Backdoor Vulnerability
- Bruce Schneier — XZ Utils Backdoor
- Black Duck (Synopsys) — Everything you need to know about the XZ Utils Backdoor
- Risky Business #743 — A chat about the xz backdoor with Andres Freund
