The internet, as anyone who works deep in its trenches will tell you, is not a smooth, well-oiled machine.
It’s a messy patchwork that has been assembled over decades, and is held together with the digital equivalent of Scotch tape and bubble gum. Much of it relies on open-source software that is thanklessly maintained by a small army of volunteer programmers who fix the bugs, patch the holes and ensure the whole rickety contraption, which is responsible for trillions of dollars in global G.D.P., keeps chugging along.
Last week, one of those programmers may have saved the internet from huge trouble.
His name is Andres Freund. He’s a 38-year-old software engineer who lives in San Francisco and works at Microsoft. His job involves developing a piece of open-source database software known as PostgreSQL, whose details would probably bore you to tears if I could explain them correctly, which I can’t.
Recently, while doing some routine maintenance, Mr. Freund inadvertently found a backdoor hidden in a piece of software that is part of the Linux operating system. The backdoor was a possible prelude to a major cyberattack that experts say could have caused enormous damage, if it had succeeded.
Now, in a twist fit for Hollywood, tech leaders and cybersecurity researchers are hailing Mr. Freund as a hero. Satya Nadella, the chief executive of Microsoft, praised his “curiosity and craftsmanship.” An admirer called him “the silverback gorilla of nerds.” Engineers have been circulating an old, famous-among-programmers web comic about how all modern digital infrastructure rests on a project maintained by some random guy in Nebraska. (In their telling, Mr. Freund is the random guy from Nebraska.)
In an interview this week, Mr. Freund — who is actually a soft-spoken, German-born coder who declined to have his photo taken for this story — said that becoming an internet folk hero had been disorienting.
“I find it very odd,” he said. “I’m a fairly private person who just sits in front of the computer and hacks on code.”
The saga began earlier this year, when Mr. Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed a few error messages he didn’t recognize. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory.
But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to log into computers remotely, was using more processing power than normal. He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to the earlier errors he’d seen.
(Don’t worry if these names are Greek to you. All you really need to know is that these are all small pieces of the Linux operating system, which is probably the most important piece of open-source software in the world. The vast majority of the world’s servers — including those used by banks, hospitals, governments and Fortune 500 companies — run on Linux, which makes its security a matter of global importance.)
Like other popular open-source software, Linux gets updated all the time, and most bugs are the result of innocent mistakes. But when Mr. Freund looked closely at the source code for xz Utils, he saw clues that it had been intentionally tampered with.
In particular, he found that someone had planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.
In the cybersecurity world, a database engineer inadvertently finding a backdoor in a core Linux feature is a little like a bakery worker who smells a freshly baked loaf of bread, senses something is off and correctly deduces that someone has tampered with the entire global yeast supply. It’s the kind of intuition that requires years of experience and obsessive attention to detail, plus a healthy dose of luck.
At first, Mr. Freund doubted his own findings. Had he really discovered a backdoor in one of the world’s most heavily scrutinized open-source programs?
“It felt surreal,” he said. “There were moments where I was like, I must have just had a bad night of sleep and had some fever dreams.”
But his digging kept turning up new evidence, and last week, Mr. Freund sent his findings to a group of open-source software developers. The news set the tech world on fire. Within hours, a fix was developed and some researchers were crediting him with preventing a potentially historic cyberattack.
“This could have been the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, the chief trust officer at SentinelOne, a cybersecurity research firm.
If it had gone undetected, Mr. Stamos said, the backdoor would have “given its creators a master key to any of the hundreds of millions of computers around the world that run SSH.” That key could have allowed them to steal private information, plant crippling malware, or cause major disruptions to infrastructure — all without being caught.
(The New York Times has sued Microsoft and its partner OpenAI on claims of copyright infringement involving artificial intelligence systems that generate text.)
Nobody knows who planted the backdoor. But the plot appears to have been so elaborate that some researchers believe only a nation with formidable hacking chops, such as Russia or China, could have attempted it.
According to some researchers who have gone back and looked at the evidence, the attacker appears to have used a pseudonym, “Jia Tan,” to suggest changes to xz Utils as far back as 2022. (Many open-source software projects are governed via hierarchy; developers suggest changes to a program’s code, then more experienced developers known as “maintainers” have to review and approve the changes.)
The attacker, using the Jia Tan name, appears to have spent several years slowly gaining the trust of other xz Utils developers and getting more control over the project, eventually becoming a maintainer, and finally inserting the code with the hidden backdoor earlier this year. (The new, compromised version of the code had been released, but was not yet in widespread use.)
Mr. Freund declined to guess who might have been behind the attack. But he said that whoever it was had been sophisticated enough to try to cover their tracks, including by adding code that made the backdoor harder to spot.
“It was very mysterious,” he said. “They clearly spent a lot of effort trying to hide what they were doing.”
Since his findings became public, Mr. Freund said, he had been helping the teams who are trying to reverse-engineer the attack and identify the culprit. But he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline.
“I don’t really have time to go and have a celebratory drink,” he said.