- It's disguised malware which they probably don't want to host (even if it's widely reported, not everyone will have heard). It's probably significantly easier to disable the repo than for them to roll it back - and even if they wanted to roll it back it's not clear where to roll back to, and introduces more confusion.
- They were probably getting a continuous stream of reports regarding the repo.
- The repo was generating a continuous stream of "lol wow" comments on every object in it regarding the situation.
- They are probably concerned about fallout towards their reputation.
- No one has a pressing need to access the repo. You can't safely use the code from it etc. I'm sure someone will upload it somewhere for study soon enough. (Preferably in such a way that it's obviously to be treated as malware)
- The most likely scenario is that momentum will gather around a fork which will be a different URL anyway...
Pulling directly from github is probably not a good thing either way. Maybe nixos should have its own mirror. Or maybe there should be an ipfs mirror that some distros and other projects could seed
Anyway I think that a new release that fixes the known vuln isn't enough. There should be a release without all contributions from the malicious dude
In this specific case this is true, but generally speaking Github can remove repositories for any reason (and for no reason). They provide free bandwidth so it gets used for that, it's just not trustworthy.
One option for distros that compile from source (such as Gentoo, NixOS without the package cache, of Arch Linux's AUR) is to make the package system to either download from Github, or download from a distro-provided mirror if Github is down. Distros could pool together to offer a source code mirror that is common for all their users. That would be a good thing.
Less puckishly: in general, as a movement/culture/economic ferment, we probably should have been more hesitant to introduce a de-facto single-point-of-failure into, like, all the things.
I mean, nothing _mandates_ that NixOS packages pull their source from github (or any other git hoster) but the standard practice is basically just this.
The nix cache will prevent the impact from this specific disablement from being, well, disabling, for most folks most of the time, for a while at least, but caching, by definition, is ephemeral storage, and sooner or later, it, ah, ephemes? Is that a word?
There are also a lot of packages that have to be built locally, e.g. nonfree packages like nvidia drivers; at least _one_ of these packages (how popular? how critical?) will probably fail to build on this account, and this in turn will cause a certain (unknown) quotient of misery. Somewhere, some lights are going out. (How many? How soon? With what mitigation?)
Anyway. Nix is obviously not the only product to rely on github as a de facto backing store, so
Yes! There are alternatives to Github, people! Diversify! Switch to Sourcehut or Gitea or Gitlab or even DIY with Gitolite. git is a decentralized peer-to-peer system, we shouldn't let Microsoft have a monopoly on its use.
> Just adding variety reduces the chance of having all the important repos up at the same time.
this, sadly, is correct, and the solution you propose is also correct.
The only other option would be for the NixOS Foundation to mirror the git repo of _every single important package_, and I know they're already struggling with the prohibitive cloud costs of hosting just the build objects. :/
I think this attitude really underrates the network effects of GitHub.
I personally am okay making an account on another forge to submit a bug report or patch, but most people aren't. And so I keep my repos on GitHub.
I do plan to soon set up mirroring to another service though. I haven't found a great zero-effort solution to mirror repos from an entire organization though, including ones that are created after mirroring was initially set up. Do you know of any tools which have that functionality? (If not I might write my own at some point.)
> openssh does not directly use liblzma. However debian and several other
distributions patch openssh to support systemd notification, and libsystemd
does depend on lzma.[0]
A quick glance at nixpkgs[1] does not show a direct dependency.
It's not a problem, never has been. Nix mirrors all source bundles it pulls from third parties and caches them. cache.nixos.org has a copy of all the sources needed to build not just current HEAD, but also past commits (although deep history might start getting pruned for cost control soon, iiuc).
In this instance then, it sounds like it has cached the source with a backdoor in it, and anyone using it is potentially exposing themselves to a very public problem right now.
Also no. It was rolled back hours ago, and cache.nixos.org keeps all past builds so it didn't even need rebuilding.
Orthogonal to that, the backdoor was irrelevant to nix in at least three different ways: the malicious build logic targeted rpm/deb build environments and so didn't trigger in nix's build sandbox, the backdoor code makes assumptions about filesystem layout that are invalid on nixos and so wouldn't have activated anyway, and nix doesn't include the downstream patch that results in the backdoor even getting into sshd's address space. Still got rolled back out of an abundance of caution, but nix got lucky that the attacker didn't bother targeting it the way they did debian and rpm-based distros.
Are they storing source archives for each version? I'd think mirroring the actual repo might take less space than a bunch of copies of source archives.
Though it looks like git only uses deflate on pack files. Someone should write a patch to add lzma support. :-)
I find it amusing that you choose to blame GitHub and not nix's brittle design. (can you imagine any other OS breaking because a repository goes down?)
Sure, but 'my updates are broken' is a happier (and more familiar to nixos-unstable users) place to be in than 'my ssh access has been compromised for days'.
Is disabling the compromised repo the typical GitHub policy? My concern is there are monorepos used by package managers, like brew, that are a collection of thousands of projects [1]. These monorepos seem like a prime target for attack and if GitHub disables one because a malicious commit was merged then you've taken down an entire ecosystem.
It’s been a few years since I worked there, but I suspect situations like this are still played by ear. They may have run books for it now, but I think human judgement is still a significant factor.
I don't think the situation is very typical, so I don't why the response should be.
There's two maintainers, one is on an "internet hiatus", the other is either suspected of being the bad actor, or is otherwise so thoroughly compromised to destroy any short-term trust.
I assume if homebrew had such an egregious issue, they'd have someone able to respond to it.
I guess it’s only temporary u until they find a new owner, but they should have made it read only to avoid breaking whatever depends on xz code to be there.
xz is also used in Windows and who knows what versions or how it is used there (impossible to verify because of the closed source nature of the windows ecosystem)
if Microsoft security detected some kind of relations between xz and exploits injected into the Windows source code/build process, they are pretty much 100% going to order github to nuke the entire repo regardless of what other people in the open source community thinks or wants.
Does anyone have a clone available for historical / analysis purposes?
I understand Microsoft doesn't want to participate in being a further attack vector - ie. "what if people use github to revert to a prior version that secretly had another vulnerability?" But removing public access to the repository prevents the community and industry analysis that would identify those prior vulnerabilities.
57 comments
[ 3.3 ms ] story [ 138 ms ] threadIt's the cause, but the logic is missing.
Makes this repo untrustworthy.
- It's disguised malware which they probably don't want to host (even if it's widely reported, not everyone will have heard). It's probably significantly easier to disable the repo than for them to roll it back - and even if they wanted to roll it back it's not clear where to roll back to, and introduces more confusion.
- They were probably getting a continuous stream of reports regarding the repo.
- The repo was generating a continuous stream of "lol wow" comments on every object in it regarding the situation.
- They are probably concerned about fallout towards their reputation.
- No one has a pressing need to access the repo. You can't safely use the code from it etc. I'm sure someone will upload it somewhere for study soon enough. (Preferably in such a way that it's obviously to be treated as malware)
- The most likely scenario is that momentum will gather around a fork which will be a different URL anyway...
Turns out one of the newer maintainers of the library managed to get a backdoor into it.
Anyway I think that a new release that fixes the known vuln isn't enough. There should be a release without all contributions from the malicious dude
One option for distros that compile from source (such as Gentoo, NixOS without the package cache, of Arch Linux's AUR) is to make the package system to either download from Github, or download from a distro-provided mirror if Github is down. Distros could pool together to offer a source code mirror that is common for all their users. That would be a good thing.
I mean, nothing _mandates_ that NixOS packages pull their source from github (or any other git hoster) but the standard practice is basically just this.
The nix cache will prevent the impact from this specific disablement from being, well, disabling, for most folks most of the time, for a while at least, but caching, by definition, is ephemeral storage, and sooner or later, it, ah, ephemes? Is that a word?
There are also a lot of packages that have to be built locally, e.g. nonfree packages like nvidia drivers; at least _one_ of these packages (how popular? how critical?) will probably fail to build on this account, and this in turn will cause a certain (unknown) quotient of misery. Somewhere, some lights are going out. (How many? How soon? With what mitigation?)
Anyway. Nix is obviously not the only product to rely on github as a de facto backing store, so
Just adding variety reduces the chance of having all the important repos up at the same time.
this, sadly, is correct, and the solution you propose is also correct.
The only other option would be for the NixOS Foundation to mirror the git repo of _every single important package_, and I know they're already struggling with the prohibitive cloud costs of hosting just the build objects. :/
I personally am okay making an account on another forge to submit a bug report or patch, but most people aren't. And so I keep my repos on GitHub.
I do plan to soon set up mirroring to another service though. I haven't found a great zero-effort solution to mirror repos from an entire organization though, including ones that are created after mirroring was initially set up. Do you know of any tools which have that functionality? (If not I might write my own at some point.)
(b) you still have to create an account there
A quick glance at nixpkgs[1] does not show a direct dependency.
0: https://www.openwall.com/lists/oss-security/2024/03/29/4
1: https://github.com/NixOS/nixpkgs/tree/master/pkgs/tools/netw...
It's not the vulnerability that's a problem right now (NixOS was protected by a couple of factors) but rather GitHub's hamfisted response.
That is the problem.
The Software Heritage archive also has an up to date mirror of xz's repo: https://archive.softwareheritage.org/browse/origin/directory...
Orthogonal to that, the backdoor was irrelevant to nix in at least three different ways: the malicious build logic targeted rpm/deb build environments and so didn't trigger in nix's build sandbox, the backdoor code makes assumptions about filesystem layout that are invalid on nixos and so wouldn't have activated anyway, and nix doesn't include the downstream patch that results in the backdoor even getting into sshd's address space. Still got rolled back out of an abundance of caution, but nix got lucky that the attacker didn't bother targeting it the way they did debian and rpm-based distros.
Debian is considering reverting prior to his involment or switching, cf: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024
Though it looks like git only uses deflate on pack files. Someone should write a patch to add lzma support. :-)
Anyway, it's called unstable for a reason. Despite debian trying to change the meaning of the term :)
Confirming no lzma linking on my recent x86_64 system: https://pastebin.com/hgN0LTxd
[1] https://github.com/Homebrew/homebrew-core
Maybe that should be an opportunity for introspection into the house of card we've collectively built as software engineers.
https://xkcd.com/2347/
There's two maintainers, one is on an "internet hiatus", the other is either suspected of being the bad actor, or is otherwise so thoroughly compromised to destroy any short-term trust.
I assume if homebrew had such an egregious issue, they'd have someone able to respond to it.
And the repo title wouldn't mention the news.
Maybe it would be allowed in this case, but a tweet or similar is safer.
So: submit the GitHub link
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
So: use what the link says, which in this case might perhaps be "xz repository has been disabled by GitHub"
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
if Microsoft security detected some kind of relations between xz and exploits injected into the Windows source code/build process, they are pretty much 100% going to order github to nuke the entire repo regardless of what other people in the open source community thinks or wants.
https://en.wikipedia.org/wiki/XZ_Utils
I understand Microsoft doesn't want to participate in being a further attack vector - ie. "what if people use github to revert to a prior version that secretly had another vulnerability?" But removing public access to the repository prevents the community and industry analysis that would identify those prior vulnerabilities.