Maybe it's time to move away from xz which has been criticized in the past as a not great format, has had maintainer issues for years, and now turns out to be infiltrated with security problems.
Moving to zstd seems worthwhile.
We could say something similar about other tools. Even something like OpenSSL had a severe security issue (Heartbleed) discovered on 2014 and which was introduced 2 years earlier by the maintainer himself through a Git commit which occurred, if I remember, on the middle of the night of 31.12.2011 or 01.01.2012 (which lead some people to speculate whether that was intentional since "everyone" is usually celebrating with his/her family at that time of the year)
It's more or less what the current last comment on the Debian thread is wondering about
"Is xz-utils going to be maintained? Will we want to keep in the archive
an unmaintained low-level library - low-level as in, susceptible of
getting pulled as a dependency in lots of places - and rely on it for
components such as dpkg?"
The format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to learn here is not about making technical improvements, it’s to do something about improving the way we manage and maintain core open source infrastructure.
No, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster.
"almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature.
If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
I will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice.
Yeah, that title is click bait. The format is perfectly fine for long term archival.
In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...
That attack article puts forward a dubious scenario where parts of multiple copies of a file become corrupted in different locations and you need to recover the original using both of the damaged copies.
If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario.
A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.
Agreed on the SSH compression, but it would be not relevant to this particular vulnerability because liblzma was never used in the SSH compression anyway.
Apparently liblzma in the compromised tarball would search for sshd functions and replace them with their own versions. This normally doesn't happen for the reason I've described, but many distros tend to compile sshd with a plugin that makes use of libsystemd, which depends on liblzma and therefore sshd will be linked with liblzma in that case. Liblzma could have done anything else it wanted, so the limited reach of this vulnerability is actually a better scenario...
The connection is that sshd calls code in libsystemd when running as a systemd service, to notify systemd that it finished starting. An unrelated functionality in libsystemd (my guess is that it's related to parsing journald logs, since they can use LZMA compression) uses the lzma library, so libsystemd is linked against it. That is, the chain is: sshd is linked against libsystemd, which is linked against liblzma. When loading the sshd executable, the dynamic linker will load all these libraries.
The backdoor was added to initialization code in liblzma (which runs whenever the dynamic linker loads that library). So that malicious code runs before sshd starts executing, and manipulates the dynamic linker so that it will do something (which is still being analyzed AFAIK) when some specific functions are called by the sshd executable.
So it is wrong for such a fundamental service as systemd to depend on such complicated processes ... just as the anti-systemd people said. Except of course shellscripts and /bin programs are just as complicated, when you check.
Migrating to zstd instead of Lzip seems an odd decision. Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger.
Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
I think it makes sense to use a commonly used format here, the maintenance aspects are very important as this incident has shown. And I don't think the tradeoffs between compressed size and speed are entirely obvious. Trading off slightly larger files for easier maintenance would seem like a defensible choice to me.
It looks like Lzip only has a single maintainer and there's no public repository to track commits or contributors (at least I couldn't find any). I don't think it's a viable option.
It doesn't even have a git/mercurial/CVS repo; just .tar.gz on some random website. And installation instructions are:
tar -xf lzip[version].tar.gz
cd lzip[version] && ./configure && make check
The entire thing seems entirely the work by one person.
Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.
Absolutely not. Extecatly this recipe brought us this desaster.
The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
> Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger.
Absolutely false because Zstandard didn't exist when xz was initially chosen. And the actual ratio difference is not that huge, generally less than 10% for most use cases, while the fastest level of xz is slower than most levels of zstd. (In fact, `dpkg-deb` uses `xz -6` by default because `-9` would be really expensive.) Given that zstd is also faster to decompress than xz as well, the ~10% increase is justifiable.
> Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
Lzip is not a better-designed format, it is just needlessly minimal for its use cases, much like xz's needless extensibility. I have detailed my own criticisms in the other comment [1].
Almost. I tested this a few years ago and IIRC zstd at its highest settings is typically about 5% larger than LZMA. So there is a difference, but probably not enough for most people to care.
I tested recently and (on that selection of data) zstd had better maximum compression at the highest level but took about 3x as long to compress at that level. At most other levels it's faster than xz with slightly worse compression, but not by much.
In what I understand: we don't know for sure, but the suspicions are so high that it more important to act quickly and revert the changes.
The community will analyse later what the changes are exactly doing ( less urgent than reverting the changes )
We definitely know for sure this was inserted intentionally.
What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.
This attack was so sophisticated that I don't understand half of it after a first read. It's plainly obvious that it is not a simple typo or a blunder but very carefully crafted.
First, the "bug" jumps through several well-crafted hurdles in order to hide itself from co-maintainers, package maintainers, and anyone else who might be looking. Hiding code in binary objects, hidden within in testcases and injected into the build under just-so conditions, would be incredibly difficult to justify with any other intent.
Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd.
Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd?
It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.
This is very much what I was thinking as well. This wasn't a casual allocation error, or some attempt to load data from a well-crafted archive to execute it or something like that.
The payload already is pretty sophisticated and full of intent to both hide itself and to target sshd very specifically. And on top, there is another layer of pretty clever build infrastructure and manipulation - again with fairly clear intent to hide the inclusion of the payload. Not to take away from the discovery, but finding this was lucky.
But then you have two actions with imo fairly clear intent to hide, deceive and target security infrastructure - and then some more mundane actions become weird as well. Deactivating parts in oss-fuzz is plausible under the right conditions, but in this light? Or being pushy to get this released into newer Fedora versions - again, might be understandable and boring in other circumstances.
Staying in your example - suddenly it becomes strange that this person got lost in the museum two or three times over the last month. Or has been taking walks around the museum very often over the last weeks. Nothing illegal with those on their own, but like that?
Cumulus Linux (originally by Cumulus Networks, which is now part of Nvidia) is a Debian-based Linux distribution used the data plane in datacenters: Dell, HPE, Mellanox, Lenovo and others let customers use Cumulus Linux. That means major datacenters running a potentially-compromised Linux distribution because of the xz attack.
McAfee Linux used to be (it EOL'ed recently) a CentOS-based distribution (which in turn comes from Fedora) used for security appliances by McAfee. Back when the xz attack was initiated (Oct'21), McAfee Linux was in full support. Even more enterprise-grade appliances to be easily compromised.
As a former Debian developer, I wonder if the "must use upstream tarball" approach Debian has (or at least, when I was a DD, just a handful of years ago) makes sense. It seems if built from the git repository, this xz attack would not have succeeded.
The malicious developer did not add bugs to xz, they used xz as an attack vector. They added malicious shell code to the xz build scripts, so that when you build and install xz it attempts to inject a backdoor into sshd. The malicious shell scripts were carefully hidden. They were compressed and added to the repository under the guise of "test files". Furthermore, the backdoor is only enabled in the release tarballs: the malicious actor has run the autoconf and then manually enabled the backdoor in the generated build scripts. If you clone the upstreaf repo and regenerate the buildscriots from scratch,the backdoor is not enabled.
When is Google, Amazon, Facebook, the DARPA, the EU - ANYONE going to step up to the plate and fund developer time for such obviously critical infrastructure? Why could a single Chinese guy undermine the entire system with so little effort?
Name choice I disagree with. The attacker used at least Chinese, Indian and Scandinavian names when working on this. It is possible we can find out something by doing a statistical analysis of the language but as far as I know no such analysis has been done.
My current hypothesis is that the attacker(s) deliberately chose a Chinese-sounding name to make people jump to conclusions. A sophisticated state actor would be no stranger to making their actions look like somebody else's.
Or maybe they actually are Chinese, and want me to believe the sentence above. :)
Indeed - seeing the level of sophistication and long term thinking behind this attack, it seems everything about it was planned meticulously, including usernames, attack vectors, communication with the community, and when to commit the parts of the exploit. There's probably a full team of people behind it and attributing it to China because of the username may be too quick.
I strongly disagree. If you scratch the surface you find ”misoeater” and some russian sounding user names, and even a danish name. These are all fake profiles and gives us no information about the bad actor.
At best there’s an account with a Chinese sounding name. I’m never going to carry water for the PLA or the CCP, but I’d need to see stronger evidence before agreeing it’s them.
If you’re implying that I used incomprehensible initialisms, I’d argue that they’re easily understandable in the context of an alleged state sponsored attack from China.
I think your comment is facetious and adds nothing to the conversation.
Occurrences of "jiat0218" on Taiwanese websites are clearly copypasta from some other website that's probably defunct now.
While gendered names do exist, you can't easily tell from the romanization, it depends on the exact characters used. And most likely "Jia" is the surname, or rather the romanization of one of five different surnames.
Tan is a common Chinese surname that's the Hokkien/Teochew transliteration of Chen. It's more commonly seen in overseas Chinese but I wouldn't jump to any conclusions
There was just 2 active maintainers on xz, and the compromised individual has been the primary for >1 year. If anyone were to fund development time on it, ironically it would go exactly to the person working to backdoor it.
Facebook would fund a Facebook engineer and would therefore have vetted the person in question. Same for other actors, there are ways to support these projects with eg a non-profit organization bankrolled by EU taxes or similar. Imagine if instead of Some Guy being the sole maintainer of critical infrastructure piece X, there was shared responsibility and accountability and less personal attachment.
We can argue about the specific implementation, but to suggest that it’s impossible seems incorrect to me.
Haven’t nation state actors openly infiltrated high level companies? This would provide a false sense of security imo. If anything, we need better testing and behavior heuristics for incoming code.
I develop cool thing X. Now, 100 minor things depend upon it.
Suddenly, Facebook (or anyone of that size!) starts using it, and decides to vet the maintainer/author.
Who says anyone has to cooperate? It's his software. He wrote it. Don't like it?
Well tough!
Now obviously Facebook could author a replacement. It could fork and maintain.
But the very nerve that Facebook(or anyone!) would insist upon a security audit of the anonymous author would be very, very strange.
Next up, I lend a neighbour my lawn mower, after he comes begging to borrow it. Oh but wait! My neighbour now wants me to sign a libabilty form, and also undergo a security check, all so he can borrow my lawnmower!
The hell?!?!
Hoping this illustrates my point. The project author owes nothing to anyone.
And it gets more wacky, if there are 100 companies demanding audits. What? Demand?!
This is where distros are the strong point. They aren't perfect, but they catch a lot of stuff on their own. And maintainers of different distros often backchannel, support each other in this.
In terms of some government org "vetting" people? Way to take the last vestiges of free software, and hacking, and turn it into a gatekeeping, bureaucratic nightmare. I guess one will need credentials, government id, a 10 year security check, to be fingerprinted, and so on? Security clearances work like that, and that's how you vet someone.
Agreed that the vetting of private people would be invasive and not a good use of resources. It would also not work for recently compromised accounts.
I’m also personally torn on how much we want giant private companies controlling more and more of the core compute infrastructure and software.
In an ideal world, the code and software itself would be automatically analyzed for malicious use cases in release and deployment pipelines, but that’s a magic hand wavy kind of ask of a huge magnitude and complexity.
This exact thing happened to me. I maintain a fairly popular free software project. A few years ago, I received an email from a nasa.gov domain claiming that they want to use the project internally and are auditing all their suppliers.
They wanted documentation on me and on how I audit my supply chain for the project. Not cool. I don't have time for these shenanigans in my personal time.
The point of the anecdote is to supplement the parent poster by stating that their hypothetical scenarios are already happening.
We should have better testing and more eyes on incoming code for projects we depend on. But my point I guess is that vetting maintainers is not an option.
And in this case Facebook did author a replacement--zstd. It just didn't get popular enough. And even when it does get popular, it won't replace all usages of xz, only some of them.
The problem is that a sole maintainer was able to shove code all the way into distros. You can vet people all you want, it won't prevent these issues as long as we all run code, which has never been looked at by a second pair of eyes.
Same reason why commercial planes always have two vetted pilots in the cockpit and ATC watching their every move. People can and will do malicious shit, if their mental health declines.
Everything looks obvious in hindsight doesn’t it? Several of the organisations you’ve mentioned have funded projects to rewrite critical internet infrastructure in a memory safe language - HTTP client, HTTP reverse proxy, DNS resolver, NTP server, TLS library, AV1 decoder, a replacement for sudo, among other projects (https://memorysafety.org). More can always be done, and it can always be done faster than it’s being done.
Fwiw, it’s possible to use zstd instead of xz, which was created at and actively maintained at Facebook.
So if you were expecting Big Tech to identify this class of problem and do something about it, it seems like they are. But if you expected them to eliminate every single possible instance of such an issue, that’s a bit much isn’t it?
Would a 'memory safe' language really have stopped any of this? From what I've briefly read, this incident the whole attack process (PR, fake test data, social engineering) from start to finish by a state-actor (most likely), could have been in any language.
There are million ways to hide exploits even in "safe" languages. If you have a bad actor that can spend couple years prepping how to hide an exploit, then it doesn't really matter which programming language is used.
Build system is complex because of legacy and need to support millions of different platforms and distros.
Different platforms and distros still share a handful of common environments, while autoconf was invented back when there were hundreds of different environments to care about. Xz doesn't depend that much on the platform and could have ditched autoconf at least in the theory. But such switch is annoying in the world of C/C++.
I'd say there are fewer ways to hide exploits like this. Rust doesn't use configure scripts. And it has a built in testing mechanism - so there's no need for makefiles to run scripts. Any build scripts as complicated as configure in a rust project would be a much bigger red flag. And it would be rare and weird to see GNU_IFUNC in use in a rust project too.
Sure; you can hide nasty code in rust. But I think it would have been harder. Not because rust is memory safe. But because rust is decades newer, and it doesn't inherit all of the quirks in C, in its build system and obscure mechanisms for dynamic linking.
Bingo. This was a highly motivated, targeted, and well planned compromise of the supply chain itself. And funnily enough, using a memory-safe language could have actually made it less likely for this to be detected.
The equivalents in memory safe languages raise more eyebrows. Segfault developing on a shared C codebase and it's a day ending with y. Segfault in a language that's meant to not let that happen? Or conversely if a patch turns up that's explicitly doing lots of unsafe things? That gets more scrutiny.
People like to harp on software written in C, and often for very good reasons. A vocal minority of those then whip out their calling card, "should have used Rust!"
As a result, there is a sentiment in the peanut gallery that "memory safe language == Rust".
You are correct in that a segfault in a project written in memory safe language gets more scrutiny. But imagine for a moment what would have happened if the compiler guaranteed that those Valgrind-caught memory access violations could not have occurred in the first place?
We got lucky, and the developers who wrote the actual backdoor were sloppy. They were aiming for sneakiness and evasion first, memory correctness would have likely been an afterthought. The next group tasked with a similar approach will know better.
You're imagining things, there is no memory corruption mechanism. The payload is delivered into the process address space by dynamic linking through a shared dependency and hijacks a function via GNU_IFUNC.
Well, Rust doesn’t support dynamic linking. So this particular attack wouldn’t be possible with Rust in March 2024.
But it’s a moot point because there are undoubtedly other mechanisms a malicious maintainer could use to inject a vulnerability. Just not this exact one.
Yes, But it would at least be harder. C has so many weird quirks and edge cases - including decades of legacy mechanisms (like IFUNC) that almost nobody today understands.
The underhanded C contest would be a lot less entertaining in rust. Thats a good thing.
It wouldn't solve the underlying problem permanently. But I don't think there's any slam-dunk solution here. I'll take any small wins where we can get them.
As far as I can tell, Rust _does_ support dynamic linking. You can both compile rust code to .so and link against existing shared objects. What are you talking about?
>The compiler supports various methods to link crates together both statically and dynamically. This section will explore the various methods to link crates together, and more information about native libraries can be found in the FFI section of the book.
Rust does support dynamic linking, it's just not that useful because of the unstable ABI. You can dynamically link against C code, or Rust code presenting a C ABI, without issue, and you can dynamically link against Rust code if you accept the inherent limitations at the moment
Well Valgrind originally went off on it and oss-fuzz presumably would have caught it since here's a PR from the attacker asking for it to be disabled: https://github.com/google/oss-fuzz/pull/10667
Right now they’re worried that somewhere in those 700 commits added by the malicious maintainer there may be some subtle memory safety bug. So great is the difficulty of finding these bugs that they’re planning to discard all work since the commiter became involved, including work by others.
Introducing such hidden bugs that are basically impossible to find is much harder in a memory safe, statically linked language.
Debian-stable users, once again, remain unaffected by $currentDumpsterFire.
(Experts: how do you weigh the risks of the two choices against each other? Running bleeding-edge upgrades brings in these fresh security holes before anyone's noticed them. But, running "stable" software can mean running vulnerable software that upstream has already patched. Not every vulnerability is recognized as a vulnerability, and patched as a security update; many "ordinary" bugs have security consequences which aren't obvious to recognize. Which choice is less worse?)
Packages in "stable" versions get bugfixes, too. How often obviously depends on the maintainer, but anything marked as a security issue upstream tends to be patched fairly quickly.
On the whole, I would assume that someone who maintains a Debian or RHEL package is better qualified to assess the severity of a bug than I am. The bigger problem is that xz hasn't had a proper maintainer for the better part of a decade.
An example that worried me was this Linux kernel bug [0], which in hindsight turned out to be a severe vulnerability, but didn't get a Debian-stable security update for three months. The vulnerability aspect wasn't obvious in the initial reports.
Is this common? Is it more common in software that has lower profile than the Linux kernel—software with fewer maintainers and fewer eyeballs, and fewer security-focused people flagging the right issues in the right ways to downstream?
Debian's development model actually seems to hit a sweet spot between "everyone runs LTS" and "everyone is cannon fodder".
Stable is LTS, sid is cutting edge, and testing is somewhere in between. You can choose the level of stability you want, and even move from one to the other, all without leaving the umbrella of Debian.
This implies that testing is "more stable" than sid, which is often not the case, if what you care about is breaking changes. Broken packages in sid tend to be updated fairly quickly, whereas breakage in testing often stays broken for quite some time until the fix is promoted.
Depends on what you mean by "stable". A distro with known-broken packages can be "stable" in a sense that its brokenness is reproducible with every new container you spin up, and therefore can be reliably worked around. I've had this happen with minor packages in known-stable distros like CentOS and Ubuntu LTS, too.
Exactly. Per discussion in the link, the safe versions are 5.2.x and earlier. So, Debian 11 (oldstable) is not affected as it runs 5.2.5, but Debian 12 (stable) could have some other backdoor from the same author.
I'm fully supportive to sandbox build.rs for that reason, but the same concern applies to pretty much everything in your computer so I'm not worried as much. In the other words, that risk is currently constant with respect to the choice of language.
It's not constant, most C projects don't have anywhere near the number of library dependencies seen in your typical cargo project. From a supply chain attack perspective, Rust and Node promote ecosystems that are total disasters.
Most software projects are not written in C. Conversely C would be used for projects where C would be less annoying than alternatives, and that condition largely precludes projects with many required dependencies. And every language-level package manager, not just that of Rust and Node, would be a disaster from the supply chain attack perspective---it can even be said that C is suffering from the huge amount of supply chain attacks in spite of the lack of good language-level package manager.
A really long con would be to first identify some component than already has a naturally occurring, so far unknown hard-to-find vulnerability, and then slowly add this kind of injected vulnerability to it. If the first vulnerability is eventually accidentally fixed by some normal refactoring, and if the second vulnerability if eventually found, everyone will revert back to the old version and now you can again use the first vulnerability.
Yes I know that one. But I don’t really see a conection still. The xz backdoor was not introduced by Debian, it was introduced by a bad faith actor via the xz project (aka liblzma). Not only Debian was affected, Fedora was too.
AFAIK there is no evidence that CVE-2008-0166 was done deliberately, whereas with the xz backdoor, there seems to be lots of signs of deliberate intent.
The backdoor was possible because Debian introduced a dependency into sshd, libsystemd, which in turn linked to liblzma. Official sshd does not have a systemd dependency in the same way that official sshd back in 2008 did not suffer from a predictable PRNG.
Debian is also 100% at fault here and I hope this slap in the face makes them reconsider their poor security practices. Amateur package maintainers should not be overstepping their job (packaging) and enabling disastrous vulnerabilities in security-critical software.
That's ridiculous. Adding the dependency to libsystemd in no way created any vulnerability. Even the indirect dependency on liblzma didn't add any vulnerability. It took a separate malicious maintainer doing a later change to liblzma so that it ran code when the library was loaded to create the vulnerability.
I wrote 'was possible' and 'enabled', not 'created'. By paying more attention to what others write you'll avoid superficial dismissals that prevent you from improving your understanding of complex issues.
The creation of this backdoor did not take place in a vacuum. Can you imagine the same person trying to introduce build changes and binary test cases into openssh? This backdoor was crafted in this particular way _specifically_ to exploit the fact that sshd in distributions such as Debian is being indirectly linked with lzma.
Never trust Devin. Should have been clear from the first Debian-specific "improvement", which brought us the first ssh mess. systemd notifications are much easier to get than linking to it, and thus enabling such a backdoor. Amateurs shouldnt mess with critical security infra, esp. Debian's
Not much. He just applied the debian-specfic systemd-notify patch to ssh to enable this xz backdoor. As he did with the previous debian ssh desaster, breaking it the last time.
It would appear that no one knows the RL identity of the bad actor? Or even vouches for them?
It doesn't seem wise to accept contributions to projects which may have security implications from anons. Code contributions clearly can't or aren't independently validated.
Contributions from anons can and in most cases will be verified by maintainers. The actual problem here is appointing them as maintainers.
EDIT: To be clear, this problem is not directed to the original xz maintainer, but more about how to prevent or reduce such appointments in the first place.
What do you propose? What would constitute a meaningful verification of identity that wouldn't create a near insurmountable hurdle for newcomers or those unable to travel?
How? AFAIK a passkey is just a replacement for a password. If my passkey tells you I can authenticate as john.doe@gmail.com, how does that help with establishing trust and identity?
In Sweden we have the proprietary BankID system, which is linked to one’s ‘personnummer’. (The personnummer is similar to SSN and compulsory for all citizens.)
Not sure if open source developers would be so keen on introducing requirements like that into the development process. Some developers want to keep a low profile and not reveal to much about their AFK identity. And nearly all tech for doing stuff like this is proprietary.
I would want to think it over but off the top of my head:
GitHub could provide this as a service, perhaps using uploads of whatever state/national ID the user is coming from. Give a little 'verified ID' badge. All banks do things like this for new accounts.
Granted this doesn't stop state-actors, but it should be effective against cyber criminals.
I think a system where it was harder for unknowns/newcomers to contribute but had better security is a good tradeoff. Esp. since it is likely this incident is not unique.
National ID or verification by an employer. Both are ubiquitous and probably strong enough to at the very least prevent a significant amount of impersonations or fake accounts, which seem to play a large part in legitimizing or disguising these attacks.
How are you going to verify a national ID? By taking a photo of it? What if the photo is edited? By somehow going to the maintainer in person and looking for anti-counterfeiting features in the paper? What if the ID is old enough that it predates these anti-counterfeiting features? Not all countries are Germany.
> or verification by an employer.
What if the maintainer is unemployed? What if the reason the maintainer has enough free time to maintain the package is because they're already retired, and living off their pension or investment returns?
> Both are ubiquitous
From what I've heard in the Internet, some large countries still don't have a national ID.
How many actually existing maintainers in particular of security sensitive parts of open source projects don't have at least some record of employment or education, a modern passport or national ID with eID features, some other comparable form of identification or at least say one reputable person in the community who can verify they are who they say they are?
Are we seriously pretending the standards that exist for ordering a beer in a bar can't be met for software security that literally billions of people depend on?
How would that help? In this case the adversary is likely to be a state actor/group with a significant amount of resources available (time, money, talent, secrecy, etc.)
Consider also that in many countries the government can force a honest national to collaborate with them on introducing backdoors to any target. This can happen both in OSS and proprietary software projects btw
That's a kneejerk reaction that will likely have more negative consequences than positive ones, and it likely wouldn't even have helped in this case. Do you think that sophisticated actors, especially state level actors, aren't capable of giving fake IDs to people? Or to have a web of equally fake identities vouch for someone?
Every project potentially has security implications. A package that's used for compression today could be linked to an SSH server tomorrow. Something as banal as a todo list could be a potential attack vector in the right hands.
I don't know what the solution to this problem is. Better verification of dependencies, certainly, but how? More investment by corporate users of FOSS into security? But no matter what we do, "who's watching the watchmen" will remain a valid question.
I'm actually quite impressed that this was found as quickly as it has, though it makes me wary that there might be other, as yet unfound, such attacks.
Your point is well-taken, but see my reply below. Stopping state-backed level actors might not be possible, but we can make it harder for cyber-criminals.
Why? Still throwing out the baby with the bathwater. Is there a huge number of problematic open source commits by "cyber criminals"? Would this even stop them? Doubtful. Will this make it harder for bona fide people to contribute? Certainly.
Good question was asked by debian devs: what's the future for xz now?
I feel it is not really worth it to vet 2 years' worth of changes by a malicious actor, who had been actively trying to make any exploits add by them as hard to discover as possible, when good alternatives, like zstd, exist.
Windows might also be affected: they just expanded built-in explorer archive format support with libarchive, which depends on one of affected libraries AFAIU: https://news.ycombinator.com/item?id=36052169
152 comments
[ 3.8 ms ] story [ 225 ms ] thread"Is xz-utils going to be maintained? Will we want to keep in the archive an unmaintained low-level library - low-level as in, susceptible of getting pulled as a dependency in lots of places - and rely on it for components such as dpkg?"
Ref XKCD 2347.
[1] https://lists.debian.org/debian-devel-announce/2011/08/msg00...
If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
[1] https://mattmahoney.net/dc/zpaq.html
In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...
If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario.
A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.
While matching LZMA2 compression ratio: no.
The backdoor was added to initialization code in liblzma (which runs whenever the dynamic linker loads that library). So that malicious code runs before sshd starts executing, and manipulates the dynamic linker so that it will do something (which is still being analyzed AFAIK) when some specific functions are called by the sshd executable.
So it is wrong for such a fundamental service as systemd to depend on such complicated processes ... just as the anti-systemd people said. Except of course shellscripts and /bin programs are just as complicated, when you check.
Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.
The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
By the xz maintainer? I very much doubt that.
Absolutely false because Zstandard didn't exist when xz was initially chosen. And the actual ratio difference is not that huge, generally less than 10% for most use cases, while the fastest level of xz is slower than most levels of zstd. (In fact, `dpkg-deb` uses `xz -6` by default because `-9` would be really expensive.) Given that zstd is also faster to decompress than xz as well, the ~10% increase is justifiable.
> Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.
Lzip is not a better-designed format, it is just needlessly minimal for its use cases, much like xz's needless extensibility. I have detailed my own criticisms in the other comment [1].
[1] https://news.ycombinator.com/item?id=39873122
What we don't know for sure is whether Jia Tan (whoever that might be, if it is even a single person) did it on purpose, or if they had their account compromised.
Next, the affected codepath skips itself if it's not executed under just-so conditions. Specifically testing to see if it's being run interactively to avoid diagnosis, and testing to see if the executed binary is sshd.
Lastly, what good reason could a compression library possibly have, for overriding encryption routines in sshd?
It's a fair question, but it doesn't survive inspection at all. This is like finding someone in a ninja suit, wearing night vision equipment, creeping around a museum in the middle of the night, with the crown jewels in a bag slung over their shoulder - and asking yourself if they're perhaps a lost tourist.
The payload already is pretty sophisticated and full of intent to both hide itself and to target sshd very specifically. And on top, there is another layer of pretty clever build infrastructure and manipulation - again with fairly clear intent to hide the inclusion of the payload. Not to take away from the discovery, but finding this was lucky.
But then you have two actions with imo fairly clear intent to hide, deceive and target security infrastructure - and then some more mundane actions become weird as well. Deactivating parts in oss-fuzz is plausible under the right conditions, but in this light? Or being pushy to get this released into newer Fedora versions - again, might be understandable and boring in other circumstances.
Staying in your example - suddenly it becomes strange that this person got lost in the museum two or three times over the last month. Or has been taking walks around the museum very often over the last weeks. Nothing illegal with those on their own, but like that?
Cumulus Linux (originally by Cumulus Networks, which is now part of Nvidia) is a Debian-based Linux distribution used the data plane in datacenters: Dell, HPE, Mellanox, Lenovo and others let customers use Cumulus Linux. That means major datacenters running a potentially-compromised Linux distribution because of the xz attack.
McAfee Linux used to be (it EOL'ed recently) a CentOS-based distribution (which in turn comes from Fedora) used for security appliances by McAfee. Back when the xz attack was initiated (Oct'21), McAfee Linux was in full support. Even more enterprise-grade appliances to be easily compromised.
As a former Debian developer, I wonder if the "must use upstream tarball" approach Debian has (or at least, when I was a DD, just a handful of years ago) makes sense. It seems if built from the git repository, this xz attack would not have succeeded.
PS: why the negatives to an honest question!???
Edit: A Russian sounding alias was also used.
Or maybe they actually are Chinese, and want me to believe the sentence above. :)
Spreading doubt is taught in bad actor-academy.
I think your comment is facetious and adds nothing to the conversation.
Neither this one, nor the previous one.
Jie would be female, Jia male
While gendered names do exist, you can't easily tell from the romanization, it depends on the exact characters used. And most likely "Jia" is the surname, or rather the romanization of one of five different surnames.
Also from that writeup. It also links to https://rheaeve.substack.com/p/xz-backdoor-times-damned-time... which suggests the actual TZ of 'Jia Tan' to be UTC+02/03
We can argue about the specific implementation, but to suggest that it’s impossible seems incorrect to me.
Haven’t nation state actors openly infiltrated high level companies? This would provide a false sense of security imo. If anything, we need better testing and behavior heuristics for incoming code.
I develop cool thing X. Now, 100 minor things depend upon it.
Suddenly, Facebook (or anyone of that size!) starts using it, and decides to vet the maintainer/author.
Who says anyone has to cooperate? It's his software. He wrote it. Don't like it?
Well tough!
Now obviously Facebook could author a replacement. It could fork and maintain.
But the very nerve that Facebook(or anyone!) would insist upon a security audit of the anonymous author would be very, very strange.
Next up, I lend a neighbour my lawn mower, after he comes begging to borrow it. Oh but wait! My neighbour now wants me to sign a libabilty form, and also undergo a security check, all so he can borrow my lawnmower!
The hell?!?!
Hoping this illustrates my point. The project author owes nothing to anyone.
And it gets more wacky, if there are 100 companies demanding audits. What? Demand?!
This is where distros are the strong point. They aren't perfect, but they catch a lot of stuff on their own. And maintainers of different distros often backchannel, support each other in this.
In terms of some government org "vetting" people? Way to take the last vestiges of free software, and hacking, and turn it into a gatekeeping, bureaucratic nightmare. I guess one will need credentials, government id, a 10 year security check, to be fingerprinted, and so on? Security clearances work like that, and that's how you vet someone.
I’m also personally torn on how much we want giant private companies controlling more and more of the core compute infrastructure and software.
In an ideal world, the code and software itself would be automatically analyzed for malicious use cases in release and deployment pipelines, but that’s a magic hand wavy kind of ask of a huge magnitude and complexity.
The point of the anecdote is to supplement the parent poster by stating that their hypothetical scenarios are already happening.
We should have better testing and more eyes on incoming code for projects we depend on. But my point I guess is that vetting maintainers is not an option.
Same reason why commercial planes always have two vetted pilots in the cockpit and ATC watching their every move. People can and will do malicious shit, if their mental health declines.
Fwiw, it’s possible to use zstd instead of xz, which was created at and actively maintained at Facebook.
So if you were expecting Big Tech to identify this class of problem and do something about it, it seems like they are. But if you expected them to eliminate every single possible instance of such an issue, that’s a bit much isn’t it?
Build system is complex because of legacy and need to support millions of different platforms and distros.
Sure; you can hide nasty code in rust. But I think it would have been harder. Not because rust is memory safe. But because rust is decades newer, and it doesn't inherit all of the quirks in C, in its build system and obscure mechanisms for dynamic linking.
Bingo. This was a highly motivated, targeted, and well planned compromise of the supply chain itself. And funnily enough, using a memory-safe language could have actually made it less likely for this to be detected.
There were symptoms showing up prior to the discovery, most prominently Valgrind errors in otherwise unrelated project builds. (ref: https://bugzilla.redhat.com/show_bug.cgi?id=2267598)
As a result, there is a sentiment in the peanut gallery that "memory safe language == Rust".
You are correct in that a segfault in a project written in memory safe language gets more scrutiny. But imagine for a moment what would have happened if the compiler guaranteed that those Valgrind-caught memory access violations could not have occurred in the first place?
We got lucky, and the developers who wrote the actual backdoor were sloppy. They were aiming for sneakiness and evasion first, memory correctness would have likely been an afterthought. The next group tasked with a similar approach will know better.
But it’s a moot point because there are undoubtedly other mechanisms a malicious maintainer could use to inject a vulnerability. Just not this exact one.
The underhanded C contest would be a lot less entertaining in rust. Thats a good thing.
It wouldn't solve the underlying problem permanently. But I don't think there's any slam-dunk solution here. I'll take any small wins where we can get them.
>The compiler supports various methods to link crates together both statically and dynamically. This section will explore the various methods to link crates together, and more information about native libraries can be found in the FFI section of the book.
https://doc.rust-lang.org/reference/linkage.html
I’m talking about what happens in practice, which is shipping static binaries that contain all of their dependencies, except for libc.
Introducing such hidden bugs that are basically impossible to find is much harder in a memory safe, statically linked language.
(Experts: how do you weigh the risks of the two choices against each other? Running bleeding-edge upgrades brings in these fresh security holes before anyone's noticed them. But, running "stable" software can mean running vulnerable software that upstream has already patched. Not every vulnerability is recognized as a vulnerability, and patched as a security update; many "ordinary" bugs have security consequences which aren't obvious to recognize. Which choice is less worse?)
On the whole, I would assume that someone who maintains a Debian or RHEL package is better qualified to assess the severity of a bug than I am. The bigger problem is that xz hasn't had a proper maintainer for the better part of a decade.
Is this common? Is it more common in software that has lower profile than the Linux kernel—software with fewer maintainers and fewer eyeballs, and fewer security-focused people flagging the right issues in the right ways to downstream?
[0] https://news.ycombinator.com/item?id=28926426
the reason $currentDumpsterFire is identified is only because it's being used. Cannon fodder exists for a purpose.
Stable is LTS, sid is cutting edge, and testing is somewhere in between. You can choose the level of stability you want, and even move from one to the other, all without leaving the umbrella of Debian.
Well, it doesn't ship systemd whatsoever, but that's a plus to me.
Also, I am a strong believer of strength by diversity, especially for anything that is safety-critical.
[1] Except when there are C/C++ dependencies :-)
It was really bad.
https://github.com/g0tmi1k/debian-ssh
AFAIK there is no evidence that CVE-2008-0166 was done deliberately, whereas with the xz backdoor, there seems to be lots of signs of deliberate intent.
Debian is also 100% at fault here and I hope this slap in the face makes them reconsider their poor security practices. Amateur package maintainers should not be overstepping their job (packaging) and enabling disastrous vulnerabilities in security-critical software.
The creation of this backdoor did not take place in a vacuum. Can you imagine the same person trying to introduce build changes and binary test cases into openssh? This backdoor was crafted in this particular way _specifically_ to exploit the fact that sshd in distributions such as Debian is being indirectly linked with lzma.
At least there is no lack of amateur enthusiasts telling people how they are wrong. :)
It doesn't seem wise to accept contributions to projects which may have security implications from anons. Code contributions clearly can't or aren't independently validated.
I also recommend this summary [1].
[0] https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h... [1] https://boehs.org/node/everything-i-know-about-the-xz-backdo...
Taking contributions from anons appears to be common. I am suggesting that should change.
EDIT: To be clear, this problem is not directed to the original xz maintainer, but more about how to prevent or reduce such appointments in the first place.
https://www.bankid.com/en/foretag/secure-digital-identificat...
I think India has something similar?
Not sure if open source developers would be so keen on introducing requirements like that into the development process. Some developers want to keep a low profile and not reveal to much about their AFK identity. And nearly all tech for doing stuff like this is proprietary.
GitHub could provide this as a service, perhaps using uploads of whatever state/national ID the user is coming from. Give a little 'verified ID' badge. All banks do things like this for new accounts.
Granted this doesn't stop state-actors, but it should be effective against cyber criminals.
I think a system where it was harder for unknowns/newcomers to contribute but had better security is a good tradeoff. Esp. since it is likely this incident is not unique.
How are you going to verify a national ID? By taking a photo of it? What if the photo is edited? By somehow going to the maintainer in person and looking for anti-counterfeiting features in the paper? What if the ID is old enough that it predates these anti-counterfeiting features? Not all countries are Germany.
> or verification by an employer.
What if the maintainer is unemployed? What if the reason the maintainer has enough free time to maintain the package is because they're already retired, and living off their pension or investment returns?
> Both are ubiquitous
From what I've heard in the Internet, some large countries still don't have a national ID.
How many actually existing maintainers in particular of security sensitive parts of open source projects don't have at least some record of employment or education, a modern passport or national ID with eID features, some other comparable form of identification or at least say one reputable person in the community who can verify they are who they say they are?
Are we seriously pretending the standards that exist for ordering a beer in a bar can't be met for software security that literally billions of people depend on?
So just claim that you are verified by your employer? It's fairly easy to setup a web page for a non-existing company...
Consider also that in many countries the government can force a honest national to collaborate with them on introducing backdoors to any target. This can happen both in OSS and proprietary software projects btw
Every project potentially has security implications. A package that's used for compression today could be linked to an SSH server tomorrow. Something as banal as a todo list could be a potential attack vector in the right hands.
I don't know what the solution to this problem is. Better verification of dependencies, certainly, but how? More investment by corporate users of FOSS into security? But no matter what we do, "who's watching the watchmen" will remain a valid question.
I'm actually quite impressed that this was found as quickly as it has, though it makes me wary that there might be other, as yet unfound, such attacks.
Has it been shown this is by a state-level actor?
Why? Still throwing out the baby with the bathwater. Is there a huge number of problematic open source commits by "cyber criminals"? Would this even stop them? Doubtful. Will this make it harder for bona fide people to contribute? Certainly.
I feel it is not really worth it to vet 2 years' worth of changes by a malicious actor, who had been actively trying to make any exploits add by them as hard to discover as possible, when good alternatives, like zstd, exist.