ifunc is a GNU method of interposing function calls with platform-optimized versions of the function. It is used to detect CPU features at runtime and insert, for example, AVX2-optimized versions of memcmp. It is seen in crypto a lot, because CPUs have many crypto-specific instructions.
However, I don't like it much and I think software should be compiled for the target machine in the first place. My 1 hardened system that is reachable from the public network is based on musl, built mostly with llvm, and with ifunc disabled.
> However, I don't like it much and I think software should be compiled for the target machine in the first place.
That means you either have to compile software locally on each machine, or you have a combinatorial explosion of possible features.
Compiling locally has several drawbacks. It needs the full compilation environment installed on every machine, which uses a lot of disk space, and some security people dislike it (because then attackers can also compile software locally on that machine); compiling needs a lot of memory and disk space, and uses a lot of processor time and electric power. It also means that signature schemes which only allow signed code cannot be used (or you need to have the signing key available on the target machine, making it somewhat pointless).
The combinatorial explosion of features has been somewhat tamed lately, by bundling sets of feature into feature levels (x86_64-v1, etc), but that still quadruples the amount of compiled code to be distributed, and newer features still have to be selected at runtime.
I don't think you can really say it is "combinatorial" because there's not a mainstream machine with AES-NI but not, say, SSSE3. In any case if there were such a machine you don't need to support it. The one guy with that box can do scratch builds.
Compiled _on_ and compiled _for_ are not the same. There must be a way to go to the target machine, get some complete dump of CPU features, copy that to the compile-box, do the build, and copy the resulting binaries back.
Obviously compiling for the target architecture is best, but for most software (things like crypto libraries excluded) 95% of the benefit of AVX2 is going to come from things like vectorized memcpy/memcmp. Building glibc using ifuncs to provide optimized implementations of these routines gives most users most of the benefit of AVX2 (or whatever other ISA extension) while still distributing binaries that work on older CPU microarchitectures.
ifunc memcpy also makes short copies suck ass on those platforms, since the dispatch cost dominates regardless of the vectorization. It's an open question whether ifunc helps or harms the performance of general use cases.
By "open question" I meant that there is compelling research indicating that GNU memcpy/memcmp is counterproductive, but the general Linux-using public did not get the memo.
On the other hand, it also means that your distro can supply a microarchitecture-specific libc and every program automatically gets the memcpy improvements. (Well, except for the golang/rust people.)
Wasn't this the point of Gentoo, back in the day? It was more about instruction scheduling and register allocation differences, but your system would be built with everything optimized for your uarch.
Interesting, I used https://ossinsight.io/analyze/JiaT75 to identify contributions from the account used by author of the backdoor. It looks like the account made other potentially problematic contributions to other projects.
The disabling of ifunc in this PR against Google's oss-fuzz project maybe one way they tried to prevent this particular backdoor being flagged by that tool? https://github.com/google/oss-fuzz/pull/10667
> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?
FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.
> One portion of the backdoor is solely in the distributed tarballs. For
easier reference, here's a link to debian's import of the tarball, but it is
also present in the tarballs for 5.6.0 and 5.6.1:
Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries
One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.
This does make me wonder how much they made a deliberate effort to build an open source portfolio so they’d look more legitimate when time came to mount an attack. It seems expensive but it’s probably not really much at the scale of an intelligence agency.
What's the salary for a software engineer in urban China? 60-80k/yr USD? Two years of that salary is cheaper than a good single shoulder fired missile. Seems like a pretty cheap attack vector to me. A Javelin is a quarter million per pop and they can only hit one target.
They are paid much less than that. However, American weapons are also far overpriced due to high labor costs, among other things. The Chinese probably have cheaper weapons.
Unless the payment is performed by foreign entity (which means a US employer is hiring a Chinese hacker), it's not a wise choice to do currency exchange when measuring salary, because it would erase other facts affecting salary, like CPI or housing price.
Apart from (both visible and invisible) taxes, I expect a senior programmer would earn ~500-700k CNY per year. Game programmers may reach up to 200k. For a team able to perform such attack, 1M/yr avg. might be reasonable.
But if this is not a state-sponsored attack, I can't find enough interest. And, if this is state-backed...contractor or some dishonest officials would a huge part, so the real cost might be >2M/yr. Considering you can get nothing during 2 year's lurking I doubt if it's feasible enough.
I am *not* a security researcher, nor a reverse engineer. There's lots of
stuff I have not analyzed and most of what I observed is purely from
observation rather than exhaustively analyzing the backdoor code.
I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.
FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).
> Not sure what I'd have done earlier in my career
To anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people
This story is an incredible testament to how open-source software can self-regulate against threats, and more broadly, it reminds us that we all stand on the shoulders of contributors like you. Thank you!
Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.
> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.
FWIW, I did a quick check on a Devuan system.
The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages.
However that stub does not link to lzma.
Huh. That's rather surprising. Do you know how MX Linux handles systemd? Devuan does that shimming of upstream. Do they perhaps just try to leave out certain packages?
Anyway. I did not see lzma in the results on Devuan running a process check (just in case).
I did see it on a Debian.
It turns out MX uses a package called systemd-shim that seems to be the Debian one:
$aptitude show systemd-shim
Package: systemd-shim
Version: 10-6
State: installed
Automatically installed: no
Priority: extra
Section: admin
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Uncompressed Size: 82.9 k
Depends: libc6 (>= 2.34), libglib2.0-0 (>= 2.39.4), cgmanager (>= 0.32)
Suggests: pm-utils
Conflicts: systemd-shim:i386
Breaks: systemd (< 209), systemd:i386 (< 209)
Description: shim for systemd
This package emulates the systemd function that are required to run the systemd helpers without using the init service
Is that protocol documented/stable? For whatever reason, daemons are choosing to link to libsystemd instead of implementing it themselves.
It doesn't matter that libsystemd links to liblzma for other reasons. It's still in the address space of any daemon that is using libsystemd for the notification protocol.
I know Golang has their own implementation of sd_notify().
For Slurm, I looked at what a PITA pulling libsystemd into our autoconf tooling would be, stumbled on the Golang implementation, and realized it's trivial to implement directly.
> I suppose that you could execute systemd-notify but that's a solution that I would not like.
What I did was to use JNA to call sd_notify() in libsystemd.so.0 (when that library exists), which works but obviously does not avoid using libsystemd. I suppose I could have done all the socket calls into glibc by hand, but doing that single call into libsystemd directly was simpler (and it can be expected to exist whenever systemd is being used).
Caveat is that golang is not a good enough actor to be a reliable indicator of whether this interface is supported, though. They’ll go to the metal because they can, not because it’s stable.
Services may be in a different mount namespace from systemd for sandboxing or other reasons (also means you have to worry about filesystem permissions I suppose). Passing an fd from the parent (systemd) is a nice direct channel between the processes
The funny thing is that libsystemd _used_ to be split into several different libraries. I certainly remember libsystemd-journal (which is presumably the part of libsystemd that pulls in liblzma) being separate to libsystemd-daemon (which is the part that implements sd_notify, as used by OpenSSH [after patching by distros]).
If that split had never happened, then liblzma wouldn't have ended up being linked into sshd...
Right, the systemd notification framework is very simple and I've used it in my projects. I didn't even know that libsystemd provided an implementation.
My Arch system was not vulnerable because openssh was not linked to xz.
IMO every single commit from JiaT75 should be reviewed and maybe even rolled back, as they have obliterated their trust.
If they hadn't been modifying SSH their users would never have been hit by this backdoor. Of course if it is actually intended to target SSH on Debian systems, the attacker would likely have picked a different dependency. But adding dependencies like Debian did here means that those dependencies aren't getting reviewed by the original authors. For security-critical software like OpenSSH such unaudited dependencies are prime targets for attacks like this.
My point was, this is not "Debian did a thing". Lots of other distros do the same thing. In this particular case, it was in fact fortunate for users of all these other distros that Debian did it, lest this vulnerability might have never been found!
Also, only users on sid (unstable) and maybe testing seem to have been affected. I doubt there are many Debian servers out there running sid.
> The script was not present in the git tree, only in the released archives.
I confess I couldn't quite figure out the branching and tagging strategy on that repo. Very weird stuff. That script seems to have been added by Sebastian Andrzej Siewior just ahead of the 5.6.0 release. It's definitely present in the Debian git tree, and probably in many other distros since others seem to be affected.
> I confess I couldn't quite figure out the branching and tagging strategy on that repo.
It's just a regular Debian packaging repository, which includes imports of upstream tarballs - nothing out of ordinary there. Debian packaging is based on tarballs, not on git repos (although in absence of upstream tarballs, Debian maintainer may create a tarball out of VCS repo themselves).
The linked repo just happens to include some tags from upstream repo, but those tags are irrelevant to the packaging. Only "debian/*" and "upstream/*" tags are relevant. Upstream VCS history is only imported for the convenience of the packager, it doesn't have to be there.
Debian's git repositories don't have any forced layout (they don't even have to exist or be up-to-date, the Debian Archive is the only source of truth - note how this repo doesn't contain the latest version of the package), but in practice most of them follow the conventions of DEP-14 implemented by gbp (in this particular case, it looks like `gbp import-orig --upstream-vcs-tag`: https://wiki.debian.org/PackagingWithGit#Upstream_import_met...).
I would phrase it as "It's good we have a heterogenous open-source community".
Monocrops are more vulnerable to disease because the same (biological) exploit works on the entire population. In our Linux biosphere where there are dozens of major, varied configurations sharing parts but not all of their code (and hundreds or thousands of minor variations), a given exploit is likely to fail somewhere, and that failure is likely to create a bug that someone can notice.
It's not foolproof, but it helps keep the ecosystem healthy.
This is what I did for a daemon I'm maintaining. Type=notify support was requested but I'm really allergic to adding new libs to a project until they really do some heavy lifting and add enough value. I was pleasantly surprised the protocol was that simple and implemented it myself. I think systemd should just provide a simple standalone reference implementation and encourage people to copy it into their project directly. (But maybe they already do, I did that almost a decade ago IIRC when the feature was relatively new.)
Whoops, you forgot `vsock:`, `@`, `SO_PASSCRED` (I think)... oh and where is that example provided? But yep that's all the protocol is for sure (and forever)!
One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.
Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.
What's systemd? 2M lines? It was >1M lines 4+ years ago.
For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.
And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.
Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.
What about sponsors? Actually, now I have the idea of a platform similar to Kickstarter but for software development, and with just sponsors. It wouldn't work, sure... Except in some cases. Like when things like this happen...
What's the point of your implementation? systemd is totally modular, you can use just the init system without networkd, timesyncd, resolved, nspawn, whatever else I forgot about.
If you want you can just use systemd as PID1 for service management and enjoy a sane way to define and manage services – and do everything in archaic ways like 20 years ago.
* Choice. If I have a separate implementation, my users do not have to be subject to systemd's choices. And I do not either.
* The same implementation will have the same bugs, so in the same way that redundant software has multiple independent implementations, having an independent implementation will avoid the same bugs. It may have different bugs, sure, but my goal would be to test like SQLite and achieve DO-178C certification. Or as close as I could, anyway.
I'd assume chances of monetizing this are incredibly low. There already is an init system that understands systemd unit files, the name escapes my mind unfortunately. DO-178C might be a selling point literally, but whether there's enough potential customers for ROI is questionable.
Well some distros might force more components upon you but thas hardly systemd's fault. Same if some software decides to make use of another component of systemd - then that's their choice, but also there are alternatives. The only thing that comes to mind right now would be something like GNOME which requires logind, but all other "typical" software only wants systemd-the-init-system if anything. You can run Debian just fine with just systemd as an init system and nothing else.
> Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on)
Most of the things you named there are modular and can be easily disabled.
Furthermore, udev precedes systemd and systemd has in fact its own replacement for it (though the name escapes me).
Kind of a classic, people loving harping on systemd without properly understanding it.
Systemd is actually pretty damn good and it's GPL licensed free software.
I understand that people don't like the way it seems to work itself into the rest of Linux user space as a dependency but that's actually our own fault for not investing the man power that Red Hat invests. We have better things to do than make our own Linux user space and so they have occupied that niche. It's free software though, we always have the freedom to do whatever we want.
By the way, all the stuff you mentioned is not really part of the actual init system, namely PID 1. There's an actual service manager for example and it's entirely separate from init. It manages services really well too, it's measurably better than all that "portable" nonsense just by virtue of using cgroups to manage processes which means it can actually supervise poorly written double forking daemons.
People are complaining that it's too big, labyrinthine, and arcane to audit, not that it doesn't work. They would prefer other things that work, but don't share those characteristics.
Also, the more extensive the remit (of this init), the more complexly interconnected the interactions between the components; the fewer people understand the architecture, the fewer people understand the code, the fewer people read the code. This creates a situation where the codebase is getting larger and larger at a rate faster than the growth of the number of man-hours being put into reading it.
This has to make it easier for people who are systemd specialists to put in (intentionally or unintentionally) backdoors and exploitable bugs that will last for years.
People keep defending systemd by talking about its UI and its features, but that completely misses the point. If systemd were replaced by something comprehensible and less internally codependent, even if the systemd UI and features were preserved, most systemd complainers would be over the moon with happiness. Red Hat invests too much into completely replacing linux subsystems, they should take a break. Maybe fix the bugs in MATE.
A shell script with a few defined arguments is not a complexly interconnected set of components. It's literally the simplest, most core, least-strongly-dependent interconnection that exists in a nix system.
Tell us you never bothered to understand how init worked before drawing a conclusion on it without telling us.
Let’s not get started on how large the kernel is. Large code bases increase attack surface, period. The only sensible solution is to micro service out the pieces and only install the bare essentials. Why does the an x86 server come with Bluetooth drivers baked in?
The kernel devs are wasting time writing one offs for every vendor known to man, and it ships to desktops too.
> Red Hat invests too much into completely replacing linux subsystems, they should take a break.
They should do whatever they feel is best for them, as should we. They're releasing free as in freedom GPL Linux software, high quality software at that. Thus I have no moral objections to their activities.
You have to realize that this is really a symptom of others not putting in the required time and effort to produce a better alternative. I know because I reinvent things regularly just because I enjoy it. People underestimate by many orders of magnitude the effort required to make something like this.
So I'm really thankful that I got systemd, despite many valid criticisms. It's a pretty good system, and it's not proprietary nonsense. I've learned to appreciate it.
Init just a more or less normal program that Linux starts by default and by convention. You can make it boot straight into bash if you want. I created a little programming language with the ultimate goal of booting Linux directly into it and bringing up the entire system from inside it.
It's just a normal process really. Two special cases that I can think of: no default signal handling, and it can't ever exit. Init will not get interrupted by signals unless it explicitly configures the signal dispositions, even SIGKILL will not kill it. Linux will panic if PID 1 ever exits so it can't do that.
Traditionally, it's also the orphaned child process reaper. Process descriptors and their IDs hang around in memory until something calls wait on them. Parent processes are supposed to do that but if they don't it's up to init to do it. Well, that's the way it works traditionally on Unix. On Linux though that's customizable with prctl and PR_SET_CHILD_SUBREAPER so you actually can factor that out to a separate process. As far as I know, systemd does just that, making it more modular and straight up better than traditional Unix, simply because this separate process won't make Linux panic if it ever crashes.
As for the service manager, this page explains process and service management extremely well:
Systemd does it right. It does everything that's described in there, does it correctly, uses powerful Linux features like cgroups for even better process management and also solves the double forking problem described in there. It's essentially a solved problem with systemd. Even the people who hate it love the unit files it uses and for good reason.
There is a secondary feature to run per-user managers, though I'm unsure whether it does run doesn't run without systemd PID1. Though it might only rely on logind.
Wow, I remember reading that PID != 1 line years ago. Had no idea they changed it. I stand corrected then. Given the existence of user service managers as well as flags like --system and --user, I inferred that they were all entirely separate processes.
Makes no sense to me why the service manager part would require running as PID 1. The maintainer just says this:
> PID 1 is very different from other processes, and we rely on that.
He doesn't really elaborate on the matter though.
Every time this topic comes up I end up searching for those so called PID 1 differences. I come up short every time aside from the two things I mentioned above. Is this information buried deep somewhere?
Just asked ChatGPT about PID 1 differences. It gave me the aforementioned two differences, completely dismissed Linux's prctl child subreaper feature "because PID 1 often assumes this role in practice" as well as some total bullshit about process group leaders and regular processes not being special enough to interact with the kernel which is just absolute nonsense.
So I really have no idea what it is about PID 1 that systemd is supposedly relying on that makes it impossible to split off the service manager from it. Everything I have read up until now suggests that it is not required, especially on Linux where you have even more control and it's not like systemd is shy about using Linux exclusive features.
> By the way, all the stuff you mentioned is not really part of the actual init system, namely PID 1
Except it literally is. I once had a systemd system suddenly refuse to boot (kernel panic because PID1 crashed or so) after a Debian upgrade, which I was able to resolve by... wait for it... making /etc/localtime not be a symlink.
Why does a failure doing something with the timezone make you unable to boot your system? What is it even doing with the timezone? What is failing about it? Who knows, good luck strace'ing PID1!
Turns out you're right and my knowledge was outdated. I seriously believed the systemd service manager was separate from its PID 1 but at some point they even changed the manuals to say that's not supported.
I was also corrected further down in the thread, with citations from the maintainers even:
As it stands I really have no idea why the service manager has not been split off from PID 1. Maintainer said that PID 1 was "different" but didn't really elaborate. Can't find much reliable information about said differences either. Do you know?
I have no idea, lol. Maybe the signal handling behavior? You can't signal PID1 (unless the process has installed its own signal handler for that signal). Even SIGKILL won't usually work.
That's my entire problem with systemd though: despite the averred modularity, it combines far too many concerns for anyone to understand how or why it works the way it does.
Yeah the signal handling thing is true, PID 1 is the only process that can handle or mask SIGKILL, maybe even SIGSTOP. The systemd manual documents its handling of a ton of signals but there's nothing in there about either of those otherwise unmaskable signals. So I don't really see how systemd is "relying" on anything. It's not handling SIGKILL, is it?
The other difference is PID 1 can't exit because Linux panics if it does. That's actually an argument for moving functionality out of PID 1.
There are other service managers out there which work outside PID 1. Systemd itself literally spawns non-PID 1 instances of itself to handle the user services. I suppose only the maintainers can tell us why they did it that way.
Maybe they are relying on the fact PID 1 traditionally reaps zombies even though Linux has a prctl for that:
What if the issue is just that nobody's bothered to write the code to move the zombie process reaping to a separate process yet? Would they accept patches in that case?
Ludicrously, that manual page straight up says systemd uses this system call to set itself up as the reaper of zombie processes:
> Some init(1) frameworks (e.g., systemd(1)) employ a subreaper process
If that's true then I really have no idea what the hell it is about PID 1 that they're relying on.
Edit: just checked the source code and it's actually true.
So they're not relying on the special signals handling and they even have special support for non-PID 1 child subreapers. Makes no sense to me. Why can't they just drop those PID == 1 checks and make a simpler PID 1 program that just spawns the real systemd service manager?
Edit: they already have a simple PID 1 in the code base!
> The other difference is PID 1 can't exit because Linux panics if it does. That's actually an argument for moving functionality out of PID 1.
I actually kinda think that can be an advantage for a service manager. If your service manager crashes an automatic reboot is nice, in a way. I doubt that's why they did it though.
> If your service manager crashes an automatic reboot is nice, in a way.
I don't think it's gonna do that! I saw it in the source code: when it's running as PID 1, systemd installs a crash handler that freezes itself in a desperate attempt to avoid the kernel panic! It's pretty amazing. They could have written it so that PID 1 watches over the service manager and just restarts it if it ever crashes. I mean, systemd already supports soft-rebooting the entire user space which is pretty much exactly what would happen if PID 1 restarted a separate service manager.
Know what else I found in the source code? Various references to /proc/1. I'm starting to think that's the true reason why they want to be PID 1...
This is a bit like complaining that the Linux kernel has 30 million lines of code, while ignoring that 3/4 of that is in hardware support (drivers) or filesystems that nobody is actually required to use at any given time.
systemd is a collection of tools, one of which is an init system. Nobody accused GNU yes of being bloated just because it's in a repository alongside 50 other tools.
yes(1) is the standard unix way of generating repeated data.
It's good to do this as quickly as possible.
I really don't understand why so many get annoyed with this code.
130 lines isn't that complicated in the scheme of things.
> that nobody is actually required to use at any given time
But that's the very problem with systemd! As time goes on you're required, whether by systemd itself or by the ecosystem around it, to use more and more of it, until it's doing not only service management but also timezones, RTC, DNS resolution, providing getpwent/getgrent, inetd, VMs and containers, bootloader, udev (without adding literally any benefit over the existing implementations), ... oh and you also have to add significant complexity in other things (like the kernel!) to use it, like namespaces (which have been a frequent source of vulnerabilities)...
> timezones, RTC, DNS resolution, providing getpwent/getgrent, inetd, VMs and containers, bootloader
How many of those are you actually required to use systemd for? At least for DNS, inetd, containers and bootloader I'm pretty sure I run a few different alternatives across my systems. I think major distros (running systemd) still ship with different dns and inetd, for containers its a lot more common to use a docker-like (probably docker or podman) than it is to use systemd-nspawn.
> oh and you also have to add significant complexity in other things (like the kernel!) to use it, like namespaces (which have been a frequent source of vulnerabilities)
Namespaces were implemented before systemd, have been used before systemd in widely used systems (for example LXC and many others). Namespaces and similar kernel features are not tied to systemd.
> How many of those are you actually required to use systemd for?
That depends on what other software you want to run, because systemd's design heavily encourages other things (distros, libraries, applications) to take dependencies on various bits. See also: every mainstream distro.
> Namespaces were implemented before systemd, have been used before systemd in widely used systems (for example LXC and many others). Namespaces and similar kernel features are not tied to systemd.
Didn't say they were. But I don't have to use LXC or many others in order to use the most popular distros and applications.
>namespaces (which have been a frequent source of vulnerabilities)...
Unprivileged user namespaces sure, but I don't think that applies to namespaces in general (which without unprivileged user namespaces can only be created by root, and LPE is the concern with unprivileged userns due to increased attack surface). systemd doesn't need unprivileged userns to run.
> One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.
this is and always has been such a dumb take.
if you'd like to implement an init (and friends) system that doesn't have "unnecessary complexity" and still provides all the functionality that people currently want, then go and do so and show us? otherwise it's just whinging about things not being like the terrible old days of init being a mass of buggy and racey shell scripts.
> about things not being like the terrible old days of init being a mass of buggy and racey shell scripts.
Zero of the major distros used System V init by default. Probably only distros like Slackware or Linux From Scratch even suggested it.
It's unfortunate that so many folks uncritically swallowed the Systemd Cabal's claims about how they were the first to do this, that, or the other.
(It's also darkly amusing to note that every service that has nontrivial pre-start or post-start configuration and/or verification requirements ends up using systemd to run at least one shell script... which is what would have often been inlined into their init script in other init systems.)
> Zero of the major distros used System V init by default. Probably only distros like Slackware or Linux From Scratch even suggested it.
I have absolutely no idea what you're trying to claim.
Are you suggesting that Debian's "sysvinit" package wasn't a System V init system? That the years I spent editing shell scripts in /etc/init.d/ wasn't System V init?
or are you making some pointless distinction about it not actually being pre-lawsuit AT&T files so it doesn't count or something?
or did you not use Linux before 2010?
if you have some important point to make, please make it more clearly.
> It's unfortunate that so many folks uncritically swallowed the Systemd Cabal's claims about how they were the first to do this, that, or the other.
I feel like you have very strong emotions about init systems that have nothing to do with the comment you're replying to.
I've been using Linux regularly since 2002. I've never regularly used a Linux that used sysvinit.
In other words, over the past ~22 years (goddamn, where did the time go?) every Linux I've regularly used has had an init system that allows you to specify service dependencies to determine their start order.
> ...Debian...
Ah. That explains it. Debian's fine to build on top of but a bad distro to actually use. (Unless you really like using five-to-ten (and in some cases 25->35) year old software that's been superseded by much-improved versions.)
You should also consider that packages named "sysvinit" sometimes aren't actually what people think of when they hear "sysvinit": <https://wiki.gentoo.org/wiki/Sysvinit>
There were plenty of those that existed even before systemd. Systemd's adoption was not a result of providing the functionality that people want but rather was a result of providing functionality that a few important people wanted and promptly took hard dependencies on.
> One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.
Complexity that would otherwise be distributed to a sea of ad-hoc shell scripts? systemd is a win
We removed tens of thousands of lines of code for fixes for those "simple" init scripts when migrating to systemd.
They are never "simple", there is always some fucking edge case, like for example we had Java apps writing its own PID few seconds after start. So any app that did start and immediately after status (like Pacemaker) threw errors.
Or how once in blue moon MySQL didn't start after reboot because it so happened that:
* the PID file from previous boot wasn't cleared
* some other app ran with same PID as it was in file
* Script did not care, script saw pid file existing and didn't start MySQL.
> so now security-critical daemons like openssh have additional dependencies like liblzma
Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?
1. systemd (at least the PID 1 part) does not talk to the network, so a remotely-accessible backdoor would need to be more complex (and thus more likely to be detected) than a backdoor that can be loaded into a listening daemon like openssh.
2. You can run Debian systems without systemd as PID 1, but you're still stuck with libsystemd because so many daemons now link with it.
Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.
But let me stress two other things:
Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.
With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.
Deferring the load of the library often just makes things harder to analyze, not necessarily more secure. I imagine many of the comments quoting `ldd` are wrongly forgetting about `dlopen`.
(I really wish there were a way to link such that the library isn't actually loaded but it still shows in the metadata, so you can get the performance benefits of doing less work but can still analyze the dependency DAG easily)
It would make things more secure in this specific backdooring case, since sshd only calls a single function of libsystemd (sd_notify) and that one would not trigger the dlopen of liblzma, hence the specific path chosen by the backdoor would not work (unless libselinux fucks it up fter all, see other comments)
Dlopen has drawbacks but also major benefits. We decided the benefits relatively clearly outweigh the drawbacks, but of course people may disagree.
I have proposed a mechanism before, that would expose the list of libs we potentially load via dlopen into an ELF section or ELF note. This could be consumed by things such as packagae managers (for auto-dep generation) and ldd. However there was no interest in getting this landed from anyone else, so I dropped it.
Note that there are various cases where people use dlopen not on hardcoded lib names, but dynamically configured ones, where this would not help. I.e. things like glibc nss or pam or anything else plugin based. But in particular pam kinda matters since that tends to be loaded into almost any kind of security relavant software, including sshd.
The plugin-based case can covered by the notion of multiple "entry points": every library that is intended to be `dlopen`ed is tagged with the name of the interface it provides, and every library that does such `dlopen`ing mentions the names of such interfaces rather than the names of libraries directly. Of course your `ldd` tool has to scan all the libraries on the system to know what might be loaded, but `ldconfig` already does that for libraries not in a private directory.
This might sound like a lot of work for a package-manager-less-language ecosystem at first, but if you consider "tag" as "exports symbol with name", it is in fact already how most C plugin systems work (a few use an incompatible per-library computed name though, or rely entirely on global constructors). So really only the loading programs need to be modified, just like the fixed-name `dlopen`.
> And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise.
That's what I think too. Do the relevant docs point this out too? Ages ago they didn't. I think we should try to avoid that people just google "implement systemd notify daemon" and end up on a page that says "link to libsystemd and call sd_notify()".
> And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise
You also need to have sshd enabled to use PAM and your sshd pam stack should include pam_selinux. Then it will be dynamically loaded only when sshd starts a PAM session.
The correct thing to do would be to put different unrelated APIs into their own library, instead of everything into libsystemd0. This has always been one of my biggest issues with it. It makes it hard to replace just one API from that library, because on a binary distribution, only one package can provide it. And as a nice side effect, surprises like this one could then be avoided.
systemd developers have already rejected that approach, so I guess we will end up with lots of reimplementations, both in individual projects and third-party libsystemd-notify style libraries.
I see that different clients implemented in different languages will need different client-libraries and maintaining all that is not something, a core project is going to do but if using the raw protocol instead of the convenience of libsystemd is a (commonly ignored) recommendation which makes a lot of sense in terms of segmentation, providing at least one reference implementation would point all systemd users into the right direction.
Recommending that each client should just implement the (trivial) protocol access itself does not make so much sense to me.
What? I don't get it? Isn't it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?
It's unfortunate that the anti-systemd party lost the war... years ago. But I don't blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn't such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).
Sometimes things evolve in ways that make us feel a little obsolete.
I've been learning NixOS for a few years now, and it would have been impossible without systemd. It's one heck of a learning curve, but when you get to the other side, you know something of great power and value. Certain kinds of complexity adds 'land' (eg. systemd) that can become 'real estate' (eg. NixOS), which in turn hopes to become 'land' for the next innovation, and so forth.
Whether this happens or not (whether it's the right kind of complexity) is really hard to assess up-front, and probably impossible without knowing the complex new technology in question very well. (And by then you have the bias of depending, in part, yourself on the success of the new tech, as you've committed significant resources to mastering it, so good luck on convincing skeptical newcomers!)
It's almost like a sort of event horizon -- once you know a complex new technology well enough to see whether or not it's useful, the conflict-of-interest makes your opinion unreliable to outsiders!
Nevertheless, the assessment process itself, while difficult to get right, is worth getting better at.
It's easy for impatience and the sensation of what I've taken to calling 'daunt' -- that intrinsic recoil that the mind has from absorbing a large amounts of information whose use case is not immediately relevant -- to dissuade one from exploring. But then, one never discovers new 'land', and one never builds new real estate!
[ Aside:
This is why I'm a little skeptical of the current rebellion against frontend frameworks. Certainly some of them, like tailwind, are clearly adding fetters to an otherwise powerful browser stack. But others, like Svelte, and to some extent, even React, bring significant benefits.
The rebellion has this vibe like, well, users _should_ prefer more simply-built interfaces, and if they don't, well, they just have bad taste. What would be more humble would be to let the marketplace (e.g. consumers) decide what is preferable, and then build that.
]
The notify protocol isn't much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler.
Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.
Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?
Just because macs don't use systemd, doesn't mean the backdoor won't work. The oss-sec post talks about liblzma having backdoors in crc32_resolve() and crc64_resolve() and that it has not been fully reversed. This could perhaps affect more than just sshd on x86-64 linux?
> Just because macs don't use systemd, doesn't mean the backdoor won't work.
Practically speaking it can't - For one the script injected into the build process tests that you're running on x86-64 linux, for another, the injected code is elf code, which wouldn't link on a mac. It also needs to manipulate dynamic linker datastructures, which would also not work the same on a mac.
> This could perhaps affect more than just sshd on x86-64 linux?
This however is true - /usr/sbin/sshd was the only argv[0] value that I found to "work", but it's possible there are others. "/usr/sbin/sshd" isn't a string directly visible in the injected code, so it's hard to tell.
The article explains numerous concurrent conditions that have to be met for the backdoor to even be activated (at build time, not runtime), which combined make it extremely unlikely this will affect SSH on macOS:
- linux
- x86-64
- building with gcc & the GNU linker
- part of a .deb or .rpm build
Add to that, as the article explains: openssh does not directly use liblzma, the only reason SSH is affected at all, is because some Linux Distros patch openssh to link it against systemd, which does depend on liblzma.
Could it affect things other than SSH on a Mac? Unlikely. The compromise was introduced in 5.6.0, but macOS Sonoma has 5.4.4 (from August last year).
That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.
You can be certain it has happened, many times. Now think of all the software we mindlessly consume via docker, language package managers, and the like.
Remember, there is no such thing as computer security. Make your decisions accordingly :)
This is my main take-away from this. We must stop using upstream configure and other "binary" scripts. Delete them all and run "autoreconf -fi" to recreate them. (Debian already does something like this I think.)
> We must stop using upstream configure and other "binary" scripts. Delete them all and run "autoreconf -fi" to recreate them.
I would go further than that: all files which are in a distributed tarball, but not on the corresponding git repository, should be treated as suspect.
Distributing these generated autotools files is a relic of times when it could not be expected that the target machine would have all the necessary development environment pieces. Nowadays, we should be able to assume that whoever wants to compile the code can also run autoconf/automake/etc to generate the build scripts from their sources.
And other than the autotools output, and perhaps a couple of other tarball build artifacts (like cargo simplifying the Cargo.toml file), there should be no difference between what is distributed and what is on the repository. I recall reading about some project to find the corresponding commit for all Rust crates and compare it with the published crate, though I can't find it right now; I don't know whether there's something similar being done for other ecosystems.
One small problem with this is that autoconf is not backwards-compatible. There are projects out there that need older autoconf than distributions ship with.
The test code generated by older autoconf is not going to be work correctly with newer GCC releases due to the deprecation of implicit int and implicit function declarations (see https://fedoraproject.org/wiki/Changes/PortingToModernC), so these projects already have to be updated to work with newer autoconf.
Typing `./configure` wont work but something like `./configure CFLAGS="-Wno-error=implicit-function-declaration"` (or whatever flag) might work (IIRC it is possible to pass flags to the compiler invocations used for checking out the existence of features) without needing to recreate it.
Also chances are you can shove that flag in some old `configure.in` and have it work with an old autoconf for years before it having to update it :-P.
Yes, it sucks to add yet another wrapper but that’s what you get for choosing non backwards compatible tools in the first place. In combination with projects that don’t keep up to date on supporting later versions.
> Why do we distribute tarballs at all? A git hash should be all thats needed...
A git hash means nothing without the repository it came from, so you'd need to distribute both. A tarball is a self-contained artifact. If I store a tarball in a CD-ROM, and look at it twenty years later, it will still have the same complete code; if I store a git hash in a CD-ROM, without storing a copy of the repository together with it, twenty years later there's a good chance that the repository is no longer available.
We could distribute the git hash together with a shallow copy of the repository (we don't actually need the history as long as the commit with its trees and blobs is there), but that's just reinventing a tarball with more steps.
(Setting aside that currently git hashes use SHA-1, which is not considered strong enough.)
except it isn't reinventing the tarball, because the git hash forces verification that every single file in the repo matches that in the release.
And git even has support for "compressed git repo in a file" or "shallow git repo in a file" or even "diffs from the last release, compressed in a file". They're called "git bundle"'s.
They're literally perfect for software distribution and archiving.
People don't know how to use git hashes, and it's not been "done". Whereas downloading tarballs and verifying hashes of the tarball has been "good enough" because the real thing it's been detecting is communication faults, not supply chain attacks.
People also like version numbers like 2.5.1 but that's not a hash, and you can only indirectly make it a hash.
> I would go further than that: all files which are in a distributed tarball, but not on the corresponding git repository, should be treated as suspect.
This and the automated A/B / diff to check the tarball against the repo, flag if mismatched.
I don't think it would help much. I work on machine learning frameworks. A lot of them(and math libraries) rely on just in time compilation. None of us has the time or expertise to inspect JIT-ed assembly code. Not even mentioning that much of the code deliberately read/write out of bound, which is not an issue if you always add some extra bytes at the end of each buffer, which could make most memory sanitizer tools useless. When you run their unit tests, you run the JIT code, then a lot of things could happen. Maybe we should ask all packaging systems splitting their build into compile and test two stages, to ensure that a testing code would not impact the binaries that are going to be published.
I would rather to read and analysis the generated code instead of the code that generates it.
Maybe it's time to dramatically simplify autoconf?
How long do we need to (pretend to) keep compatibility with pre-ANSI C compilers, broken shells on exotic retro-unixes, and running scripts that check how many bits are in a byte?
Autoconf is m4 macros and Bourne shell. Most mainstream programming languages have a packaging system that lets you invoke a shell script. This attack is a reminder to keep your shell scripts clean. Don't treat them as an afterthought.
I'm wondering is there i.e. no way to add an automated flagging system that A/B / `diff` checks the tarball contents against the repo's files and warns if there's a mismatch? This would be on i.e. GitHub's end so that there'd be this sort of automated integrity test and subsequent warning? Just a thought, since tainted tarballs like these might be altogether be (and become) a threat vector, regardless of the repo.
Not just autoconf. Build systems in general are a bad abstraction, which leads to lots and lots of code to try to make them do what you want. It's a sad reality of the mismatch between a prodecural task (compile files X, Y, and Z into binary A) and what we want (compile some random subset of files X, Y, and Z, doing an arbitrary number of other tasks first, into binary B).
For fun, you can read the responses to my musing that maybe build systems aren't needed: https://news.ycombinator.com/item?id=35474996 (People can't imagine programming without a build system - it's sad)
It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.
Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?
> "Given the activity over several weeks, the committer is either directly
involved or there was some quite severe compromise of their
system. Unfortunately the latter looks like the less likely explanation, given
they communicated on various lists about the "fixes" mentioned above."
A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.
How many people read autoconf scripts, though? I think those filters are symptom of the larger problem that many popular C/C++ codebases have these gigantic build files which even experts try to avoid dealing with. I know why we have them but it does seem like something which might be worth reconsidering now that the tool chain is considerably more stable than it was in the 80s and 90s.
The alternatives are _better_ but still not great. build.rs is much easier to read and audit, for example, but it’s definitely still the case that people probably skim past it. I know that the Rust community has been working on things like build sandboxing and I’d expect efforts to be a lot easier there than in a mess of m4/sh where everyone is afraid to break 4 decades of prior usage.
build.rs is easier to read, but it's the tip of the iceberg when it comes to auditing.
If I were to sneak in some underhanded code, I'd do it through either a dependency that is used by build.rs (not unlike what was done for xz) or a crate purporting to implement a very useful procedural macro...
I mean, autoconf is basically a set of template programs for snffing out whether a system has X symbol available to the linker. Any replacement for it would end up morphing into it over time.
We have much better tools now and much simpler support matrices, though. When this stuff was created, you had more processor architectures, compilers, operating systems, etc. and they were all much worse in terms of features and compatibility. Any C codebase in the 90s was half #ifdef blocks with comments like “DGUX lies about supporting X” or “SCO implemented Y but without option Z so we use Q instead”.
Even in binary you can see patterns. Not saying it's perfect to show binary diffs (but it is better than showing nothing) but I know even my slow mammalian brain can spot obvious human readable characters in various binary encoding formats. If I see a few in a row which doesn't make sense, why wouldn't I poke it?
This particular file was described as an archive file with corrupted data somewhere in the middle. Assuming you wanted to scroll that far through a hexdump of it, there could be pretty much any data in there without being suspicious.
testdata should not be on the same machine as the build is done. testdata (and tests generally) aren't as well audited, and therefore shouldn't be allowed to leak into the finished product.
Sure - you want to test stuff, but that can be done with a special "test build" in it's own VM.
You're right - the two exploit files are lzma-compressed and then deliberately corrupted using `tr`, so a hex dump wouldn't show anything immediately suspicious to a reviewer.
Is this lzma compressed? Hard to tell because of the lack of formatting, but this looks like amd64 shellcode to me.
But that's not really important to the point - I'm not looking at a diff of every committed favicon.ico or ttf font or a binary test file to make sure it doesn't contain a shellcode.
in this case the backdoor was hidden in a nesting doll of compressed data manipulated with head/tail and tr, even replacing byte ranges inbetween. Would've been impossible to find if you were just looking at the test fixtures.
The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.
A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".
Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".
Unfortunately eval in a shell script has an effect on the semantics but is not necessary to do some kind of parsing of the contents of a variable, unlike Python or Perl or JavaScript. A
$goo
line (without quotes) will already do word splitting, though it won't do another layer of variable expansion and unquoting, for which you'll need
This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?
Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)
NixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).
Right, but in this case it's not even compiled it, which is arguably better than compiled in but assumed dormant :) (at least until someone actually does a full analysis of the payload).
Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.
That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.
It's actually not an advantage. The reason why the exploit wasn't included is because the attacker specifically decided to only inject x86_64 Debian and RHEL to reduce the chances of this getting detected.
Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.
Note that Fedora 40 isn't even released yet, it's in beta, Fedora 41 / rawhide is basically a development branch used only by a small number of people.
i knew there was an advantage to being 8-10 years out of date at all times...
and when they do port finally backport this bug in 2026, they will probably implement the systemd integration with openssl (pbthththt...) via 600 patch files in some nonstandard divergent manner that thwarts the payload anyhow. see? i knew they were super duper secure.
So many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.
I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.
This needs a Rust joke. You know, the problem with the whole certification charade is it slows down jobs and prevents __actual_problems getting evaluated. But is it safe?
> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
That is actually a major point of a lot of corporate security measures (shifting risk)
If you installed xz on macOS using brew, then you have
xz (XZ Utils) 5.6.1
liblzma 5.6.1
which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with
Yeah it was when I posted the comment too. That's why you could type brew upgrade xz and it went back to 5.4.6 I guess? But it might have been around that time, cutting it fine, not out for everybody. I don't know. Comment race condition haha! :)
Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.
Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.
Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).
What you describe is a normal process in order to minimise damage from attacks. The damage of hacking is ultimately property damage. The procedures you've described allow you to minimise it.
That's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.
I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.
[root@archlinux ~]# pacman -Qi xz | head -n2
Name : xz
Version : 5.6.1-2
[root@archlinux ~]# pacman -Qi openssh | head -n2
Name : openssh
Version : 9.7p1-1
[root@archlinux ~]# ldd $(which sshd) | grep liblzma
[root@archlinux ~]#
Interesting, they just switched from tarballs to source 19 hours ago. It seems to me that Frederik Schwan had prior knowledge of the security issue, or it is just a rare coincidence.
On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.
This is what the `detect_sh.bin` attached to the email does. I can only assume that the pesron who reported the vulnerability checked that this succeeds in detecting it.
Note that I'm not looking for the vulnerable symbols, I'm looking for the library that does the patching in the first place.
5.6.1-2 is not an attempted fix, it's just some tweaks to Arch's own build script to improve reproducibility. Arch's build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb/RPM based distros, so a narrow miss for Arch here.
(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)
Are you sure about that? The diff moves away from using the compromised tarballs to the not-compromised (by this) git source. The comment message says it's about reproducibility, but especially combined with the timing it looks to me like that was just to avoid breaking an embargo.
So, you suggest that Frederik Schwan had prior knowledge of the security issues but hid the real purpose of the commit under "improve reproducibility"?
And, If you break the embargo too many times then you just find out with the rest of us and that's not a great way to run a distro. I believe openbsd is or was in that position around the time of the intel speculative execution bugs.
xz was masked in the Gentoo repositories earlier today with the stated reason of "Investigating serious bug". No mention of security. It's pretty likely.
The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.
I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.
Do you have a writeup or any details as to what it does? The logical thing based on this post is that it hooks the SSH key verification mechanism to silently allow some attacker-controlled keys but I wonder if there's more to it?
I was starting one, but the openwall message linked here is far more detailed and gets much further than I did. It's fiendishly difficult to follow the exploit.
sshd starts with root privileges and then proceeds to, in summary:[1]
1. Parse command line arguments
2. Setup logging
3. Load configuration files
4. Load keys/certificates into memory (notably including private keys)
5. Listen on a socket/port for incoming connections
6. Spawn a child process with reduced permissions (on Linux, using seccomp filters [2]) to respond to each incoming connection request
This backdoor executes at order 0 before sshd's main function is invoked, overwriting internal sshd functions with compromised ones. As some ideas of what the backdoor could achieve:
1. Leak server private keys during handshakes with users (including unauthenticated users) allowing the keys to be passively stolen
2. Accept backdoor keys as legitimate credentials
3. Compromise random number generation to disable perfect forward secrecy
4. Execute code on the host (supplied remotely by a malicious user) with the 'root' permissions available to sshd upon launch. On most Linux distributions, systemd-analyze security sshd.service will give a woeful score of 9.6/10 (10 being the worst).[3] There is essentially NO sandboxing used because an assumption is made that you'd want to login as root with sshd (or sudo/su to root) and thus would not want to be restricted in what filesystem paths and system calls your remote shell can then invoke.
The same attacker has also added code to Linux kernel build scripts which causes xz to be executed (xz at this point has a backdoor compiled into it) during the build of the Linux kernel where xz compression is used for the resulting image. Using this approach, the attacker can selectively choose to modify certain (or all) Linux kernel builds to do some very nasty things:
1. Leak Wireguard keys allowing them to be passively intercepted.
2. Compromise random number generation, meaning keys may be generated with minimal entropy (see Debian certificate problem from a few years ago).
3. Write LUKS master keys (keys used by dm-crypt for actually decrypting disks) to disks in retrievable format.
4, Introduce remote root code execution vulnerabilities into basic networking features such as TCP/IP code paths.
I guess it seems like the operational parts are a bit poorly done. Valgrind issues, adding a new version with symbols removed, the aforementioned performance issues. Like i would assume the type of person who would do this sort of thing, over a 2 year period no less, would test extensively and be sure all their i's are dotted. Its all kind of surprising given how audacious the attack is.
There are so many variations of Linux/FreeBSD and weird setups and environments that it's almost guaranteed that you'll hit a snag somewhere if you do any major modification like inserting a backdoor.
It's hard enough to get code to work correctly; getting it to be also doing something else is even harder.
The way they went around it, however, was brilliant. Completely reduce the variables to directly target whatever it is you're attacking. Reminds me of stuxnet somewhat.
Note that in this case the backdoor was only inserted in some tarballs and enabled itself only when building deb/rpm packages for x86-64 linux and with gcc and the gnu linker. This should already filter out the most exotic setups and makes it harder to reproduce.
But they almost got away with it. We could have found ourselves 5 years later with this code in all stable distribution versions, IoT devices etc.
Also, we only catch the ones that we ... catch. The ones that do everything perfectly, unless they come out and confess eventually, we don't get to "praise" them for their impeccable work.
Sure, however the problem that software is really hard also impacts bad actors. So it's probably at least as hard to write that one line logic bug and have it do exactly what you intended as to write equivalent real code that works precisely as intended.
Pointing dogs (bird dogs) are made to point in the direction where they have perceived game. Good dogs are then not distracted by anything and stand there motionless, sometimes so far that they have to be carried away because they cannot turn away themselves.
Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.
He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
Not sure why are people downvoting you... it's pretty unlikely that various Chinese IoT companies would just decide it's cool to add a backdoor, which clearly implies that no matter how good their intentions are, they simply might have no other choice.
There are roughly speaking two possibilities here:
1. His machine was compromised, he wasn't at fault past having less than ideal security (a sin we are all guilty of). His country or origin/residence is of no importance and doxing him isn't fair to him.
2. This account was malicious. There's no reason we should believe that the identity behind wasn't fabricated. The country of origin/residence is likely falsified.
In neither case is trying to investigate who he is on a public forum likely to be productive. In both cases there's risk of aiming an internet mob at some innocent person who was 'set up'.
The back door is in the upstream GitHub tarball. The most obvious way to get stuff there is by compromising an old style GitHub token.
The new style GitHub tokens are much better but it’s somewhat intransparent what options you need. Most people also don’t use expiring tokens.
The authors seems to have a lot of oss contributions, so probably an easy target to choose.
I think the letters+numbers naming scheme for both the main account and the sockpuppets used to get him access to xz and the versions into distros is a strong hint at (2). Taking over xz maintainership without any history of open source contributions is also suspicious.
But my point is that people living in China might be "forced" to do such things, so we unfortunately can't ignore the country. Of course, practically this is problematic since the country can be faked
They might have burnt the reputation built for this particular pseudonym but what is stopping them from doing it again? They were clearly in it for the long run.
I literally said "they", I know, I know, in English that can also be interpreted as a gender unspecific singular.
Anyways, yes it is an interesting question whether he/she is alone or they are a group. Conway's law probably applies here as well. And my hunch in general is that these criminal mad minds operate individually / alone. Maybe they are hired by an agency but I don't count that as a group effort.
1. You don't actually know what has been done by whom or why. You don't know if the author intended all of this, or if their account was compromised. You don't know if someone is pretending to be someone else. You don't know if this person was being blackmailed, forced against their will, etc. You don't really know much of anything, except a backdoor was introduced by somebody.
2. Assuming the author did do something maliciously, relying on personal reputation is bad security practice. The majority of successful security attacks come from insiders. You have to trust insiders, because someone has to get work done, and you don't know who's an insider attacker until they are found out. It's therefore a best security practice to limit access, provide audit logs, sign artifacts, etc, so you can trace back where an incursion happened, identify poisoned artifacts, remove them, etc. Just saying "let's ostracize Phil and hope this never happens again" doesn't work.
3. A lot of today's famous and important security researchers were, at one time or another, absolute dirtbags who did bad things. Human beings are fallible. But human beings can also grow and change. Nobody wants to listen to reason or compassion when their blood is up, so nobody wants to hear this right now. But that's why it needs to be said now. If someone is found guilty beyond a reasonable doubt (that's really the important part...), then name and shame, sure, shame can work wonders. But at some point people need to be given another chance.
100% fair -- we don't know if their account was compromised or if they meant to do this intentionally.
If it were me I'd be doing damage control to clear my name if my account was hacked and abused in this manner.
Otherwise if I was doing this knowing full well what would happen then full, complete defederation of me and my ability to contribute to anything ever again should commence -- the open source world is too open to such attacks where things are developed by people who assume good faith actors.
I think I went through all the stages of grief. Now at the stage of acceptance here’s what I hope: I hope justice is done. Whoever is doing this be they a misguided current black hat (hopefully, future white hat) hacker, or just someone or someones that want to see the world burn or something in between that we see justice. And then forgiveness and acceptance and all that can happen later.
Mitnick reformed after he was convicted (whether you think that was warranted or not). Here if these folks are Mitnick’s or bad actors etc let’s get all the facts on the table and figure this out.
What’s clear is that we all need to be ever vigilant: that seemingly innocent patch could be part of a more nefarious thing.
We’ve seen it before with that university sending patches to the kernel to “test” how well the core team was at security and how well that went over.
Anyways. Yeah. Glad you all allowed me to grow. And I learned that I have an emotional connection to open source for better or worse: so much of my life professional and otherwise is enabled by it and so threats to it I guess I take personally.
It is reasonable to consider all commits introduced by the backdoor author untrustworthy. This doesn't mean all of it is backdoored, but if they were capable of introducing this backdoor, their code needs scrutiny. I don't care why they did it, whether it's a state-sponsored attack, a long game that was supposed to end with selling a backdoor for all Linux machines out there for bazillions of dollars, or blackmail — this is a serious incident that should eliminate them from open-source contributions and the xz project.
There is no requirement to use your real name when contributing to open source projects. The name of the backdoor author ("Jia Tan") might be fake. If it isn't, and if somehow they are found to be innocent (which I doubt, looking at the evidence throughout the thread), they can create a new account with a new fake identity.
Every Linux box inside AWS, Azure, and GCP and other cloud providers that retains the default admin sudo-able user (e.g., “ec2”) and is running ssh on port 22.
I bet they intended for their back door to eventually be merged into the base Amazon Linux image.
my understanding is that any Debian/RPM-based Linux running sshd would become vulnerable in a year or two. The best equivalent of this exploit is the One Ring.
So the really strange thing is why they put so little effort into making this undetectable. All they needed was to make it use less time to check each login attempt.
In the other hand it was very hard to detect. The slow login time was the only thing that gave it away. It more seems like they were so close to being highly successful. In retrospect improving the performance would have been the smart play. But that is one part that went wrong compared to very many that went right.
You don't need a "ec2" user. A backdoor can just allow root login even when that is disabled for people not using the backdoor.
It just requires the SSH port to be reachable unless there is also a callout function (which is risky as people might see the traffic).
And with Debian and Fedora covered and the change eventually making its way into Ubuntu and RHEL pretty much everything would have this backdoor.
Distro build hosts and distro package maintainers might not be a bad guess. Depends on whether getting this shipped was the final goal. It might have been just the beginning, part of some bootstrapping.
I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...
There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.
The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...
Also seeing this bug. Extra valgrind output causes some failed tests for me. Looks like the new version will resolve it. Would like this new version so I can continue work.
That looks exactly like what you'd want to see to disguise the actual request you want, a number of pointless upstream updates in things that are mostly ignored, and then the one you want.
Wow, what a big pile of infrastructure for a non-optimization.
An internal call via ifunc is not magic — it’s just a call via the GOT or PLT, which boils down to function pointers. An internal call through a hidden visibility function pointer (the right way to do this) is also a function pointer.
The even better solution is a plain old if statement, which implements the very very fancy “devirtualization” optimization, and the result will be effectively predicted on most CPUs and is not subject to the whole pile of issue that retpolines are needed to work around.
Right, IFUNCs make sense for library function where you have the function pointer indirection anyway. Makes much less sense for internal functions - only argument over a regular function pointer would be the pointer being marked RO after it is resolved (if the library was linked with -z relro -z now), but an if avoids even that issue.
Jia Tan getting maintainer access looks like it is almost certainly to be part of the operation. Lasse Colling mentioned multiple times how Jia has helped off-list and to me it seems like Jia befriended Lasse as well (see how Lasse talks about them in 2023).
Also the pattern of astroturfing dates back to 2022. See for example this thread where Jia, who has helped at this point for a few weeks, posts a patch, and a <name><number>@protonmail (jigarkumar17) user pops up and then bumps the thread three times(!) lamenting the slowness of the project and pushing for Jia to get commit access: https://www.mail-archive.com/xz-devel@tukaani.org/msg00553.h...
Naturally, like in the other instances of this happening, this user only appears once on the internet.
It’s certainly a pseudonym just like all the other personas we’ve seen popping up on the mailing list supporting this “Jia Tan” in these couple of years. For all intents and purposes they can be of any nationality until we know more.
PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get:
xz --version
Homebrew has already taken action, a `brew upgrade` will downgrade back to the last known good version.
GitHub disabled the xz repo, making it a bit more difficult for nix to revert to an older version. They've made a fix, but it will take several more days for the build systems to finish rebuilding the ~220,000 packages that depend on the bootstrap utils.
What should they be relying on instead? Maybe rsync everything to an FTP server? Or Torrents? From your other comments, you seem to think no one should ever use GitHub for anything.
Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…
Is that version truly vetted? "Jia Tan" has been the official maintainer since 5.4.3, could have pushed code under any other pseudonym, and controls the signing keys. I would have felt better about reverting farther back, xz hasn't had any breaking changes for a long time.
"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.
I’ve long thought that those “this new version fixes bugs and improves user experience” patch notes that Meta et al copy and paste on every release shouldn’t be permitted.
Tell me about it. I look at all these random updates that get pushed to my mobile phone and they all pretty much have that kind of fluff in the description. Apple/Android should take some steps to improve this or outright ban this practice. In terms of importance to them though I imagine this is pretty low on the list.
I have dreamed about an automated LLM system that can "diff" the changes out of the binary and provide some insight. You know give back a tiny bit of power to the user. I'll keep dreaming.
It's worse, as someone who does try to privide release notes I'm often cut off by the max length of the field. And even then, Play only shows you the notes for the latest version of the app.
Ugh. That's especially annoying because they're trying to be hip with slang and use a metaphor that requires cultural knowledge that you can't really assume everyone has.
// The "-2" is included because the for-loop will
// always increment by 2. In this case, we want to
// skip an extra 2 bytes since we used 4 bytes
// of input.
i += 4 - 2;
I don't know how you can be missing the essence of the problem here or that comments point.
Vague claims are meaningless and valueless and are now even worse than that, they are a red flag.
Please don't tell me that you would accept a pr that didn't explain what it did, and why it did it, and how it did it, with code that actually matched up with the claim, and was all actually something you wanted or agreed was a good change to your project.
Updating to the next version of a library is completely unrelated. When you update a library, you don't know what all the changes were to the library, _but the librarys maintainers do_, and you essentially trust that librarys maintainers to be doing their job not accepting random patches that might do anything.
Updating a dependency and trusting a project to be sane is entirely a different prospect from accepting a pr and just trusting that the submitter only did things that are both well intentioned and well executed.
If you don't get this then I for sure will not be using or trusting your library.
Yeah. Had a weird problem last week where GitHub was serving old source code from the raw url when using curl, but showing the latest source when coming from a browser.
Super frustrating when trying to develop automation. :(
These shouldn't be suspended, and neither should their repositories. People might want to dig through the source code. It's okay if they add a warning on the repository, but suspending _everything_ is a stupid thing to do.
This can also be handled relatively easily. They can disable the old links and a new one can be added specifically for the disabled repository. Or even just let the repository be browsable through the interface at least.
Simply showing one giant page saying "This respository is disabled" is not helpful in any way.
Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...
I agree on principle, but sometimes programmatic generating test data is not so easy.
E.g.: I have a specific JPEG committed into a repository because it triggers a specific issue when reading its metadata. It's not just _random_ data, but specific bogus data.
But yeah, if the test blob is purely random, then you can just commit a seed and generate in during tests.
EDIT: Lasse Collin's account @Larhzu has also been suspended.
EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.
--
EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn't appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:
EDIT: Just sent an email to the last committer to bring it to their attention.
EDIT: It's been removed.
--
jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.
-NickServ- Information on jiatan (account jiatan):
-NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
-NickServ- Last seen : (less than two weeks ago)
-NickServ- User seen : (less than two weeks ago)
-NickServ- Flags : HideMail, Private
-NickServ- jiatan has enabled nick protection
-NickServ- *** End of Info ***
/whowas expired not too long ago, unfortunately. If anyone has it I'd love to know.
They are not registered on freenode.
EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).
Just for posterity since I can no longer edit: Libera staff has been firm and unrelenting in their position not to disclose anything whatsoever about the account. I obtained the last point on my own. Libera has made it clear they will not budge on this topic, which I applaud and respect. They were not involved whatsoever in ascertaining a VPN was used, and since that fact makes anything else about the connection information moot, there's nothing else to say about it.
Respect not giving out identifying information on individuals whenever someone asks, no matter what company they work for and what job they do? Yes. I respect this.
I am not LE nor a government official. I did not present a warrant of any kind. I asked in a channel about it. Libera refused to provide information. Libera respecting the privacy of users is of course something I applaud and respect. Why wouldn't I?
The alpine patch includes gettext-dev which is likely also exploited as the same authors have been pushing gettext to projects where their changes have been questioned
It's still the wrong way to go about things. Tests are there for a reason, meaning if they fail you should try to understand them to the point where you can fix the problem (broken test or actual bug) instead of just wantonly distabling tests until you get a green light.
Good question, though I can imagine they took this action for two reasons:
1. They don't have the ability to freeze repos (i.e. would require some engineering effort to implement it), as I've never seen them do that before.
2. Many distros (and I assume many enterprises) were still linking to the GitHub releases to source the infected tarballs for building. Disabling the repo prevents that.
The infected tarballs and repo are still available elsewhere for researchers to find, too.
They could always archive it. Theoretically (and I mean theoretically only), there's another reason for Microsoft to prevent access to repo: if a nation state was involved, and there've been backdoor conversations to obfuscate the trail.
Archiving the repo doesn't stop the downloads. They would need to rename it in order to prevent distro CI/CD from keeping downloading untrustworthy stuff.
The latest commit is interesting (f9cf4c05edd14, "Fix sabotaged Landlock sandbox check").
It looks like one of Jia Tan's commits (328c52da8a2) added a stray "." character to a piece of C code that was part of a check for sandboxing support, which I guess would cause the code to fail to compile, causing the check to fail, causing the sandboxing to be disabled.
If your project becomes complex enough eventually you need tests for the configure step. Even without malicious actors its easy to miss that a compiler or system change broke some check.
Asking this here too: why isn't there an automated A/B or diff match for the tarball contents to match the repo, auto-flag with a warning if that happens? Am I missing something here?
The tarballs mismatching from the git tree is a feature, not a bug. Projects that use submodules may want to include these and projects using autoconf may want to generate and include the configure script.
> Those days are pretty much behind us. Sure, you can compile code and tweak software configurations if you want to--but most of the time, users don't want to. Organizations generally don't want to, they want to rely on certified products that they can vet for their environment and get support for. This is why enterprise open source exists. Users and organizations count on vendors to turn upstreams into coherent downstream products that meet their needs.
> In turn, vendors like Red Hat learn from customer requests and feedback about what features they need and want. That, then, benefits the upstream project in the form of new features and bugfixes, etc., and ultimately finds its way into products and the cycle continues.
"and when the upstream is tainted, everyone drinks poisoned water downstream, simple as that!"
> The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
Multiple suggestions on that thread on how that's a legacy practice that might be outdated, especially in the current climate of cyber threats.
After reading the original post by Andres Freund, https://www.openwall.com/lists/oss-security/2024/03/29/4, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?
That account seems to be a contributor for xz though, you can see him interact a lot with the author of the backdoor on the GitHub repo. Some pull requests seem to be just the two of them discussing and merging stuff (which is normal but looks weird in this context)
You can quite easily generate a realistic photo, bio, even entire personal blogs and GitHub projects, using generative AI, to make it look like it's a real person.
Even if they have a "real" picture or a credible description that is not good enough. Instead of using an anime character a malicious actor could use an image generator [0], they could generate a few images, obtain something credible to most folks, and use that to get a few fake identities going. Sadly, trusting people to be the real thing and not a fake identity on the Internet is difficult now and it will get worse.
Just FYI, krygorin4545@proton.me (the latest message before the upload) was created Tue Mar 26 18:30:02 UTC 2024, about an hour earlier than the message was posted.
Proton generates PGP key upon creating the account, with the real datetime of the key (but the key does not include the timezone).
That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person
It's not ironic, this change is really sinister IMO. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).
Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.
Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.
One thing that is annoying is that many open source projects have been getting "garbage commits" apparently from people looking to "build cred" for resumes or such.
Easier and easier to hide this junk in amongst them.
I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.
This is the sort of case where america's over the top hacking laws make sense.
Maybe I'm miss understanding things, but it seems like anyone can publish an exploit on the internet without being a crime. In the same way encryption is free speech.
It would seem unlikely this guy would be also logging into peoples boxes after this.
It seems a much tougher job to link something like this to an intentional unauthorized access.
At this point, we have no confirmed access via compromise.
Do you know of a specific case where the existence of a backdoor has been prosecuted without a compromise?
Who would have standing to bring this case? Anyone with a vulnerable machine? Someone with a known unauthorized access. Other maintainers of the repo?
IANAL but it is unclear that a provable crime has been committed here
It's not worth your time or the reader's time trying to come up with a technicality to make it perfectly legal to do something we know little about, other than it's extremely dangerous.
Law isn't code, you gotta violate some pretty bedrock principles to pull off something like this and get away with it.
Yes, if you were just a security researcher experimenting on GitHub, it's common sense you should get away with it*, and yes, it's hard to define a logical proof that ensnares this person, and not the researcher.
* and yes, we can come up with another hypothetical where the security researcher shouldn't get away with it. Hypotheticals all the way down.
1. It should be legal to develop or host pen-testing/cracking/fuzzing/security software that can break other software or break into systems. It should be illegal to _use_ the software to gain _unauthorised_ access to others' systems. (e.g. it's legal to create or own lockpicks and use them on your own locks, or locks you've been given permission to pick. It's not legal to gain unauthorised access _using_ lockpicks)
2. It should be illegal to develop malware that _automatically_ gains unauthorised access to systems (trojans, viruses, etc.). However, it should be legal to maintain an archive of malware, limiting access to vetted researchers, so that it can be studied, reverse-engineered and combatted. (e.g. it's illegal to develop or spread a bioweapon, but it's ok for authorised people to maintain samples of a bioweapon in order to provide antidotes or discover what properties it has)
3. What happened today: It should be illegal to intentionally undermine the security of a project by making bad-faith contributions to it that misrepresent what they do... even if you're a security researcher. It could only possibly be allowed done if an agreement was reached in advance with the project leaders to allow such intentional weakness-probing, with a plan to reveal the deception and treachery.
Remember when university researchers tried to find if LKML submissions could be gamed? They didn't tell the Linux kernel maintainers they were doing that. When the Linux kernel maintainers found out, they banned the entire university from making contributions and removed everything they'd done.
No, people being polite and avoiding the more direct answer that'd make people feel bad.
The rest of us understand that intuitively, and that it is already the case, so pretending there was some need to work through it, at best, validates a misconception for one individual.
Less important, as it's mere annoyance rather than infohazard: it's wildly off-topic. Legal hypotheticals where a security researcher released "rm -rf *" on GitHub and ended up in legal trouble is 5 steps downfield even in this situation, and it is a completely different situation. Doubly so when everyone has to "IANAL" through the hypotheticals.
> but it seems like anyone can publish an exploit on the internet without being a crime
Of course. The mere publishing of the exploit is not the criminal part. Its the manner & intent in which it was published that is the problem.
> At this point, we have no confirmed access via compromise.
While i don't know the specifics for this particular law, generally it doesn't matter what you actually did. What is relavent is what you tried to do. Lack of success doesn't make you innocent.
> Who would have standing to bring this case?
The state obviously. This is a criminal matter not a civil one. You don't even need the victim's consent to bring a case.
By this logic you could say that leaving a poisoned can of food in a public pantry is not a crime because poison is legal for academic purposes, and whoever ate it took it willingly.
Also, I think getting malicious code into a repo counts as a compromise in and of itself.
Well, it is much easier said than done. Philosophically I agree, but in the real world where you have later commits that might break and downstream projects, etc, it isn't very practical. It strikes me as in a similar vein to high school students and beauty pageant constestants calling for world peace. Really great goal, not super easy to implement.
I would definitely be looking at every single commit though and if it isn't obviously safe I'd be drilling in.
Some of those commits might fix genuine vulnerabilities. So you might trade a new backdoor for an old vulnerability that thousands of criminal orgs have bots for exploiting.
Damage wise, most orgs aren't going to be hurt much by NSA or the Chinese equivalent getting access, but a Nigerian criminal gang? They're far more likely to encrypt all your files and demand a ransom.
Still.. At this point the default assumption should be every commit is a vulnerability or facilitating a potential vulnerability.
For example, change from safe_fprintf to fprintf. It would be appropriate that every commit should be reviewed and either tweaked or re-written to ensure the task is being done in the safest way and doesn't have anything that is "off" or introducing a deviation from the way that codebase standardly goes about tasks within functions.
randomly reverting two years of things across dozens of repositories will break them, almost definitely make them unbuildable, but also make them unreleasable in case any other change needs to happen soon.
all of their code needs to be audited to prove it shouldn't be deleted, of course, but that can't happen in the next ten minutes.
I swear that HN has the least-thought-through hot takes of any media in the world.
Hoe will you do that practically though? That’s probably thousands of commits upon which tens or hundred thousand commits from others were built. You can’t just rollback everything two years and expect it not to break or bring back older vulnerabilities that were patched in those commits.
Not really. xz worked fine 2 years ago. Roll back to 5.3.1 and apply a fix for the 1 security hole that was fixed since that old version. (ZDI-CAN-16587)
Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.
(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)
Yes, it's usually regenerated already. However even the source is often pretty gnarly.
And in general, the build system of a large project is doing a lot of work and is considered pretty uninteresting and obscure. Random CMake macros or shell scripts would be just as likely to host bad code.
This is also why I like meson, because it's much more constrained than the others and the build system tends to be more modular and the complex parts split across multiple smaller, mostly independent scripts (written in Python or bash, 20-30 lines max). It's still complex, but I find it easier to organize.
> And in general, the build system of a large project is doing a lot of work and is considered pretty uninteresting and obscure. Random CMake macros or shell scripts would be just as likely to host bad code.
Build systems can even have undefined behaviour in the C++ sense. For example Conan 2 has a whole page on that.
The other thing besides the autoconf soup is the XZ project contains incomprehensible binaries as "test data"; the "bad-3-corrupt_lzma2.xz"
part of the backdoor that they even put in the repo.
It's entirely possible they could have got that injection through review, even if they had that framwork and instead put it in source files used to generate autoconf soup.
gradle-wrapper is just a convenience, you can always just build the project with an installed version of gradle.
Although I get your point, it’s a great place to hide nefarious code.
Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.
many people are patriots of their countries. if state agency would approach them proposing to have paid OSS work and help their country to fight terrorism/dictatorships/capitalists/whatever-they-believe, they will feel like killing two birds with one job
While this seems plausible, it is notable that this person seems to be anonymous from the get go. Most open source maintainers are proud of their work and maintain publicly available personas.
While I don't doubt there are people who would gladly do this work for money/patriotism/whatever, adding a backdoor to your own project isn't really reconcilable with the motivations behind wanting to do OSS work.
Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.
For those panicking, here are some key things to look for, based on the writeup:
- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.
- A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.
- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.
Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.
It is possible there are other flaws or backdoors in liblzma5, though.
I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.
So for 5.4.5 the tagged release and download on github differ.
It does change format strings, e.g.
+#: src/xz/args.c:735
+#, fuzzy
+#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
+msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
+msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"
There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.
FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code.
Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?
I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?
My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.
> I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?
Possible but unlikely.
> I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?
It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.
The article gives a link to a simple shell script that detects the signature of the compromised function.
> Running OpenSSH sshd from systemd
I think this is irrelevant.
From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
I think the distributions that do use systemd are the ones that add the libsystemd code, which in turn brings in the liblzma5 code. So, it may not be entirely relevant how it is run, but it needs to be a version of OpenSSH patched.
> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively.
However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context.
> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary.
> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps.
I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.
> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.
.deb has epochs too, but I think Debian developers avoid it where possible because 1:5.4.5 is interpreted as newer than anything without a colon, so it would break eg. packages that depend on liblzma >= 5.0, < 6. There may be more common cases that aren't coming to mind now.
Seems like debian is mixing too many things into the package version - version used for deciding on upgrades and abi version for dependencies should be decoupled like it is in modern RPM distros.
If a binary library ABI is backwards-incompatible, they change the package name. I was just guessing at the reason epoch is avoided and that <6 is probably an awful example.
So now I actually bothered to look it up, and it turns out the actual reason is that the epoch changes what version is considered "greater", but it's not part of the .deb filename, so you still can't reuse version numbers used in the past. If you release 5.0, then 5.1, then you want to rollback and release 1:5.0, it's going to break things in the Debian archives. https://www.debian.org/doc/debian-policy/ch-binary.html#uniq...
Debian has epochs, but it's a bad idea to use them for this purpose.
Two reasons:
1. Once you bump the epoch, you have to use it forever.
2. The deb filename often doesn't contain the epoch (we use a colon which isn't valid on many filesystems), so an epoch-revert will give the same file name as pre-epoch, which breaks your repository.
So, the current best practice is the +really+ thing.
Honestly, the Gentoo-style global blacklist (package.mask) to force a downgrade is probably a better approach for cases like this. Epochs only make sense if your upstream is insane and does not follow a consistent numbering system.
Gentoo also considers the repository (+overlays) to be the entire set of possible versions so simply removing the bad version will cause a downgrade, unlike debian and RPM systems where installing packages outside a repository is supported.
Stop the cap your honor. There is not a single filesystem that prevents you from using colons in filenames except exfat, I went ahead and checked and ext4, xfs, btrfs, zfs, and even reiserfs let you use any characters you want except \0 and /.
And I fail to see why bumping the epoch would ever be a problem. Using the epoch not a reason why its bad.
I really like the XBPS way of the reverts keyword in the package template that forces a downgrade from said software version. It's simple but works without any of the troubles RPM epochs have with resolving dependencies as it's just literally a way to tell xbps-install that "yeah, this is a lower version number in the repository but you should update anyway".
> If you're not on a rolling release distro, your version is probably older.
Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).
Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.
From what I'm understanding it's trying to patch itself into the symbol resolution step of ld.so specifically for libcrypto under systemd on x86_64. Am I misreading the report?
That's a strong indication it's targeting sshd specifically.
Lots of software links both liblzma and libcrypto. As I read Andres Freund's report, there is still a lot of uncertainty:
"There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code."
The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:
Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness
Most people, to find the affected versions, would either have to bisect or delve deep enough to find the offending commit. Either of which would reveal the attacker.
By not asking for the version, there is a good chance you just report "It's acting oddly, plz investigate".
I think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
Yikes indeed. This fix is being rolled out very fast, but what about the entire rest of the codebase? And scripts? I mean, years of access? I'd trust no aspect of this code until a full audit is done, at least of every patch this author contributed.
(note: not referring to fedora here, a current fix is required. But just generally. As in, everyone is rolling out this fix, but... I mean, this codebase is poison in my eyes without a solid audit)
edit: I have to be missing something, or I'm confused. The above author seems to be primary contact for xz? Have they just taken over?? Or did the bad commit come from another source, and a legit person applied it?
It's important to focus on people, not just code, when suspecting an adversary. Now, I have no idea if this is the right account, and if it has recently been compromised/sold/lost, or if it has always been under the ownership of the person who committed the backdoor. But IF this is indeed the right account, then it's important to block any further commit from it to any project, no matter how innocuous it seems, and to review thoroughly any past commit. For the most security-conscious projects, it would be a good idea to even consider reverting and re-implementing any work coming from this account if it's not fully understood.
An account that has introduced a backdoor is not the same thing as an account who committed a bug.
They appear to have moved carefully to set this up over the course of weeks by setting up the framework to perform this attack.
I would now presume this person to be a hostile actor and their contributions anywhere and everywhere must be audited. I would not wait for them to cry 'but my bother did it', because an actual malicious actor would say the same thing. The 'mob' should be pouring over everything they've touched.
My above post shows the primary domain for xz moving from tukaani.org to xz.tukaani.org. While it's hosted on github:
$ host xz.tukaani.org
host xz.tukaani.org is an alias for tukaani-project.github.io.
And originally it was not:
$ host tukaani.org
tukaani.org has address 5.44.245.25
(seemingly in Finland)
It was moved there in Jan of this year, as per the commit listed in my prior post. By this same person/account. This means that instead of Lasse Collin's more restrictive webpage, an account directly under the control of the untrusted account, is now able to edit the webpage without anyone else's involvement.
For example, to make subtle changes in where to report security issues to, and so on.
So far I don't see anything nefarious, but at the same time, isn't this the domain/page hosting bad tarballs too?
Zoner is a Finnish web hosting company, which has a history of providing hosting for Finnish open source projects, and the original maintainer (and most of the original crew) is Finnish as well. Nothing weird here.
For what it's worth, tukaani is how you spell toucan (the bird) in Finnish, and Lasse is a common Finnish name; the site being previously hosted in Finland is very plausible.
Yeah according to their website[0] it looks like majority of the past contributors were Finnish so nothing odd about the hosting provider. On the same page it says that Jia Tan became co-maintainer of xz in 2022.
This account changed the instructions for reporting security issues in the xz github as their very last commit:
commit af071ef7702debef4f1d324616a0137a5001c14c (HEAD -> master, origin/master, origin/HEAD)
Author: Jia Tan <jiat0218@gmail.com>
Date: Tue Mar 26 01:50:02 2024 +0800
Docs: Simplify SECURITY.md.
diff --git a/.github/SECURITY.md b/.github/SECURITY.md
index e9b3458a..9ddfe8e9 100644
--- a/.github/SECURITY.md
+++ b/.github/SECURITY.md
@@ -16,13 +16,7 @@ the chance that the exploit will be used before a patch is released.
You may submit a report by emailing us at
[xz@tukaani.org](mailto:xz@tukaani.org), or through
[Security Advisories](https://github.com/tukaani-project/xz/security/advisories/new).
-While both options are available, we prefer email. In any case, please
-provide a clear description of the vulnerability including:
-
-- Affected versions of XZ Utils
-- Estimated severity (low, moderate, high, critical)
-- Steps to recreate the vulnerability
-- All relevant files (core dumps, build logs, input files, etc.)
+While both options are available, we prefer email.
This project is maintained by a team of volunteers on a reasonable-effort
basis. As such, please give us 90 days to work on a fix before
Seems innocuous, but maybe they were planning further changes.
If the owner of the account is innocent and their account was compromised, it's on them to come out and say that. All signs currently point to the person being a malicious actor, so I'll proceed on that assumption.
Probably not. I did some pattern of life analysis on their email/other identifiers. It looks exactly like when I set up a burner online identity- just enough to get past platform registration, but they didn't care enough to make it look real.
For example, their email is only registered to GitHub and Twitter. They haven't even logged into their Google account for almost a year. There's also no history of it being in any data breaches (because they never use it).
It would be interesting to hear the whole arc of social engineering behind getting access to the repo. Although, as a maintainer of a large-ish OSS project myself, I know that under a lot of burden any help will be welcomed with open arms, and I've never really talked about private stuff with any of them.
Unlike the GNU tar I'm used to, it's actually a "full fat" command line archiving tool, compressing & decompressing zip, xz, bz2 on the command-line - really handy :-O
No. There's no good reason HTTP response decoding would ever be implemented in terms of libarchive; using libz directly is simpler and supports some use cases (like streaming reads) which libarchive doesn't.
EDIT: Ahh, I was wrong and missed the addition of "strerror"
The PR is pretty devious.
JiaT75 claims is "Added the error text when printing out warning and errors in bsdtar when untaring. Previously, there were cryptic error messages" and cites this as fixing a previous issue.
The PR literally removes a new line between 2 arguments on the first `safe_fprintf()` call, and converts the `safe_fprintf()` to unsafe direct calls to `fprintf()`. In all cases, the arguments to these functions are exactly the same! So it doesn't actually make the error messages any different, it doesn't actually solve the issue it references. And the maintainer accepted it with no comments!
It does remove the safe prefixes... But it also adds one print statement to "strerror()", which could plausibly give better explanations for the error code...
The only suspicious thing here is the lack for safe_ prefix (and the potential for the strerror() function to already be backdoored elsewhere in another commit)
That seems to be fine. safe_fprintf() takes care of non-printable characters. It's used for archive_entry_pathname, which can contain them, while "unsafe" fprintf is used to print out archive_error_string, which is a library-provided error string, and strerror(errno) from libc.
We know there's long-cons in action here, though. This PR needn't be the exploit. It needn't be anywhere _temporally_ close to the exploit. It could just be laying groundwork for later pull requests by potentially different accounts.
Exactly. If we assume the backdoor via liblzma as a template, this could be a ploy to hook/detour both fprintf and strerror in a similar way. Get it to diffuse into systems that rely on libarchive in their package managers.
When the trap is in place deploy a crafted package file that appears invalid on the surface level triggers this trap. In that moment fetch the payload from the (already opened) archive file descriptor, execute it, but also patch the internal state of libarchive so that it will process the rest of the archive file as if nothing happened, and the desired outcome also appearing in the system.
Assuming there isn't another commit somewhere modifying a library-provided error string or anything returned by libc. There is all kinds of mischief to be had there, which may or may not have already happened, e.g. now you do some i18n and introduce Unicode shenanigans.
I don't know whether this is a formerly-legitimate open source contributor who went rogue, or a deep-cover persona spreading innocuous-looking documentation changes around to other projects as a smokescreen.
Minor documentation change PRs is a well known tactic used to make your GitHub profile look better (especially to potential employers).
He could be doing the same thing for other reasons; nobody really digs into anything very deep so I could see someone handing over co-maintenance to a project based on a decent looking Github graph and some reasonability.
Consider the possibility those type of submissions were part of the adversary's strategy in order to make their account appear more legitimate rather than appearing out of nowhere wanting to become the maintainer of some project.
additionally, even though the commit messages they've made are mostly plain, there may be features of their commit messages that could provide leads, such as his using what looks like a very obscure racist joke of referring to a gitignore file as a 'gitnigore'.
There's barely a handful of people on the whole planet making this 'joke'.
If I wanted to go rogue and insert a backdoor in a project of mine, I'd probably create a new sockpuppet account and hand over management of the project to them. The above is worringly compatible with this hypothesis.
OTOH, JiaT75 did not reuse the existing hosting provider, but rather switched the site to github.io and uploaded there old tarballs:
If JiaT75 is an old-timer in the project, wouldn't they have kept using the same hosting infra?
There are also some other grim possibilities: someone forced Lasse to hand over the project (violence or blackmailing? as farfetched as that sounds)... or maybe stole Lasse devices (and identity?) and now Lasse is incapacitated?
Or maybe it's just some other fellow scandinavian who pretended to be chinese and got Lasse's trust. In which case I wish Lasse all the best, and hope they'll be able to clear their name.
Is the same person sockpuppeting Hans Jansen? It's amusing (but unsurprising) that they are using both german-sounding and chinese-sounding identities.
That said, I don't think it's unreasonable to think that Lasse genuinely trusted JiaT75, genuinely believed that the ifunc stuff was reasonable (it probably isn't: https://news.ycombinator.com/item?id=39869538 ) and handed over the project to them.
And at the end of the day, the only thing linking JiaT75 to a nordic identity is a nordic racist joke which could well be a typo. People already checked the timezone of the commits, but I wonder if anyone has already checked the time-of-day of those commits... does it actually match the working hours that a person genuinely living (and sleeping) in China would follow? (of course, that's also easy to manipulate, but maybe they could've slip up)
Anyhow, I guess that security folks at Microsoft and Google (because of JiaT75 email account) are probably going to cooperate with authorities on trying to pin down the identity of JiaT75 (which might not be very useful, depending on where they live).
I tried to understand the significance of this (parent maybe implied that they reused a completely fictitious identity generated by some test code), and I think this is benign.
That project just includes some metadata about a bunch of sample projects, and it links directly to a mirror of the xz project itself:
I assume it downloads the project, examines the git history, and the test then ensures that the correct author name and email addresses are recognized.
(that said, I haven't checked the rest of the project, so I don't know if the code from xz is then subsequently built, and or if this other project could use that in an unsafe manner)
I'd say at this point all major tech companies, ISPs and authorities should have more enough information and disabling and freezing their accounts would be the first step.
This can happen if you delete your old gmail account. Source: I deleted a gmail account I shouldn't have years ago. It will say taken if it previously existed, and was deleted.
There is also a variety of new, parallelized implementations of compression algorithms which would be good to have a close look at. Bugs causing undefined behaviour in parallel code are notoriously hard to see, and the parallel versions (which are actually much faster) could be take the place of well-established programs which have earned a lot of trust.
> Versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key . The older releases have been signed with Lasse Collin's OpenPGP key .
It must be assume that before acquiring that privilege, they also contributed code to project. Probably most was to establish respectable record. Still could be malicious code going back someways.
I mention Chinese social media specifically because I know it's not indexed so well by western search engines. You can't conclude someone has no social footprint until you've actually checked.
Regardless of how likely you think it is, finding a social media footprint would be useful information. Seek information first, reach conclusions second.
I think it's not a coincidence: Hans Jansen (hansjansen162@outlook.com) has a matching account on Proton mail too (hansjansen162@proton.me).
Furthermore, the Outlook account is configured as recovery e-mail for the Proton account.
Keep in mind that having a "false identity" does not make you a malicious actor. I have a serious project I work on under another pseudonym, but it has to do more with the fact that I do not want my real name to be associated with that project AND having a serious case of impostor syndrome. :/
That, and I used to contribute to various games (forks of ioquake3) when I was a teen and I wanted to keep my real name private.
I am more interest about his git commits https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...
If JiaT75 is a Chinese, then his working log should follow Chinese Holiday, especially Spring Festival and National Holiday.
Chinese usually not work on first 3 days of Spring Festival and National Holiday -
2021 2/11 - 2/13 (few commits), 2021 10/1 - 10/3 (nothing)
2022 1/31 - 2/2 (huge commits on 1/31, suspect), 2022 10/1 - 10/3 (nothing)
2023 and 2024, not very much commits.
So 2022 1/31 huge commits is a proof that he is not follow Chinese holiday.
But wait, 2021 is his active year, but he missed almost all Aug. Is he on holiday? Who can have such a long holiday? What i can think is a solider who has a long vacation (探亲假).
So let's guess he is a solider then it's sense that he worked on Spring Holiday because they need on duty.
Let's double check again, if he is a solider, then they will have a holiday on every Aug. 1 because it's liberation army day. I check and no commits on all 4 years Aug. 1.
I get why people are focusing on this bad actor. But the question that interests me more: how many other apparent individuals fit the profile that this person presented before caught?
Are you referencing the '-unsafe' suffix in the second link? That is not something to worry about.
This is from Gnulib, which is used by Gettext and other GNU projects. Using 'setlocale (0, NULL)' is not thread-safe on all platforms. Gnulib has modules to work around this, but not all projects want the extra locking. Hence the name '-unsafe'. :)
I would presume it's a state actor. Generally in the blackhat world, attackers have very precise targets. They want to attack this company or this group of individuals. But someone who backdoors such a core piece of open source infrastructure wants to cast a wide net to attack as many as possible. So that fits the profile of a government intelligence agency who is interested in surveilling, well, everything.
Or it could in theory be malware authors (ransomware, etc). However these guys tend to aim at the low hanging fruits. They want to make a buck quickly. I don't think they have the patience and persistence to infiltrate an open source project for 2 long years to finally gain enough trust and access to backdoor it. On the other hand, a state actor is in for the long term, so they would spend that much time (and more) to accomplish that.
So that's my guess: Jia Tan is an employee of some intelligence agency. He chose to present an asian persona, but that's not necessarily who he truly represents. Could be anyone, really: Russia, China, Israel, or even the US, etc.
Edit: given that Lasse Collin was the only maintainer of xz utils in 2022 before Jia Tan, I wouldn't be surprised if the state actor interfered with Lasse somehow. They could have done anything to distract him from the project: introduce a mistress in his life, give him a high-paying job, make his spouse sick so he has to care for her, etc. With Lasse not having as many hours to spend on the project, he would have been more likely to give access to a developer who shows up around the same time and who is highly motivated to contribute code. I would be interested to talk to Lasse to understand his circumstances around 2022.
Or they have just one or a small number of targets, but don’t want the target(s) to know that they were the only target(s), so they backdoor a large number of victims to “hide in the crowd”.
I agree that this is likely a state actor, or at least a very large & wealthy private actor who can play the long game…
Don't forget that you could have state actors who are otherwise interested in open source code, and working to actually improve it.
In fact, that'd be the best form of deep cover. It'll be interested to watch as people more knowledgable than I pour over every single commit and change.
If you have a backdoor in a specific piece of software already, what is the purpose of trying to introduce another backdoor (and risk it getting caught)?
There are two general attack targets I'd use if I had access to a library/binary like xz:
(1) A backdoor like this one, which isn't really about its core functions, but about the fact that it's a library linked into critical code, so that you can use it to backdoor _other things_. Those are complex and tricky because you have to manipulate the linking/GOT specifically for a target.
(2) Insert an exploitable flaw such as a buffer overflow so that you can craft malicious .xz files that result in a target executing code if they process your file. This is a slightly more generic attack vector but that requires a click/download/action.
Not every machine or person you want to compromise has an exposed service like ssh, and not every target will download/decompress a file you send to them. These are decently orthogonal attack vectors even though they both involve a library.
(Note that there's as yet no evidence for #2 - I'm just noting how I'd try to leverage this to maximum effect if I wanted to.)
xz is a data compression tool, so it's natural to have compressed files for (de)compression tests.
these files are also useful to check that the library we just built works correctly. but they aren't necessary for installation.
we may have more sophisticated procedures that will allow us to use some parts of distribution only for tests. This may significantly reduce an attack vector - many projects have huge, sophisticated testing infrastructure where you can hide the entire Wikipedia.
> I wouldn't be surprised if the state actor interfered with Lasse somehow
People could also just get tired after years of active maintainership or become busier with life. Being the sole maintainer of an active open source project on top of work and perhaps family takes either a lot of enthusiasm or a lot of commitment. It's not really a given that people want to (or can) keep doing that forever at the same pace.
Someone then spots the opportunity.
I have no idea what the story is here but it might be something rather mundane.
> I haven't lost interest but my ability to care has been fairly limited
mostly due to longterm mental health issues but also due to some other
things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and
perhaps he will have a bigger role in the future, we'll see.
That "Jigar Kumar" is like fake and one-time throw-off account, probably from the same state actor to orchestrate the painstakingly prepared supply chain attack (under the sun).
At first glance I thought it was a far-fetched conclusion but then I read in a subsequent reply he wrote:
> With your current rate, I very doubt to see 5.4.0 release this year. The only
progress since april has been small changes to test code. You ignore the many
patches bit rotting away on this mailing list. Right now you choke your repo.
Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?
Oh wow, all his posts are trying to pressure Lasse, or guilt him into getting Jia on board. They're definitely conspiring.
"Your efforts are good but based on the slow release schedule it will unfortunatly be years until the community actually gets this quality of life feature."
"Patches spend years on this mailing list. 5.2.0 release was 7 years ago. There is no reason to think anything is coming soon."
"With your current rate, I very doubt to see 5.4.0 release this year. The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"
"Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this."
"Is there any progress on this? Jia I see you have recent commits. Why can't you commit this yourself?"
"Over 1 month and no closer to being merged. Not a suprise."
> They want to attack this company or this group of individuals. But someone who backdoors such a core piece of open source infrastructure wants to cast a wide net to attack as many as possible.
The stuxnet malware, which compromised Siemens industrial controls to attack specific centrifuges in uranium enrichment plants in Iran, is a counterexample to that.
Stuxnet wasn't similar to this xz backdoor. The Stuxnet creators researched (or acquired) four Windows zero-days, a relatively short-term endeavor. Whereas the xz backdoor was a long-term 2.5 years operation to slowly gain trust from Lasse Collin.
But, anyway, I'm sure we can find other counter-examples.
If a government wants to cast a wide nest and catch what they can, they'll just throw a tap in some IXP.
If a government went to this much effort to plant this vulnerability, they absolutely have targets in mind - just like they did when they went to the effort of researching (or acquiring) four separate Windows zero-days, combining them, and delivering them...
It's ridiculous to think it's the US as it would be an attack on Red Hat a US company and an attack on Americans. It's a good way to be dragged in front of Congress.
The US has backdoored RSA's RNG and thus endangered the security of American companies. It is naive to think that US intelligence agencies will act in the best interest of US citizens or companies.
What type of confirmation do you want? The documents aren't going to be declassified in the next couple of decades, if ever.
I've never heard anyone claim that Dual_EC_DRBG is most likely not intentionally backdoored, but there's literally no way to confirm because of how its written. If we can't analyze intention from the code, we can look at the broader context for clues. The NSA spent an unusual amount of effort trying to push forward an algorithm that kept getting shot down because it was slower than similar algorithms with no additional benefits (the $10 million deal specified it as a requirement [1]). If you give the NSA the benefit of the doubt, they spent a lot of time and money to... intentionally slow down random number generation?!
As an American, I'd prefer a competent NSA than an incompetent NSA that spends my tax dollars to make technology worse for literally no benefit...
You are understating the level of evidence that points to the NSA being fully aware of what it was doing.
To be clear, the method of attack was something that had been described in a paper years earlier, the NSA literally had a program (BULLRUN) around compromising and attacking encryption, and there were security researchers at NIST and other places that raised concerns even before it was implemented as a standard. Oh, and the NSA paid the RSA $10 million to implement it.
Heck, even the chairman of the RSA implies they got used by the NSA:
In an impassioned speech, Coveillo said RSA, like many in industry, has worked with the NSA on projects. But in the case of the NSA-developed algorithm which he didn’t directly name, Coviello told conference attendees that RSA feels NSA exploited its position of trust. In its job, NSA plays two roles, he pointed out. In the information assurance directorate (IAD) arm of NSA, it decides on security technologies that might find use in the government, especially the military. The other side of the NSA is tasked with vacuuming up data for cyber-espionage purposes and now is prepared to take an offensive role in cyber-attacks and cyberwar.
“We can’t be sure which part of the NSA we’re working with,” said Coviello with a tone of anguish. He implied that if the NSA induced RSA to include a secret backdoor in any RSA product, it happened without RSA’s consent or awareness.
There's an implicit "always" in their second sentence, if you're confused by the wording. They aren't positing the equivalent of the guard that only lies.
It's an interesting story for those who haven't heard about that an think the NSA could only be up to evil. You may not have read it as the guard only ever lies, but that doesn't stop people from thinking that anyway.
No I think that's it. "What about it?" kinda set me off, and then "if you're confused by the wording" was unnecessarily condescending.
You coulda just pointed out that just because they did right in the case of DSA, doesn't mean we should actually ever trust them, which I would agree is the correct stance.
Mostly I think that story is neat and wanted people to know about it, so I asked a question as a performative writing technique.
"What about it?" is a very real question that I still want to know the answer to. What did you want as a response when you asked that?
"If you're confused by the wording" was definitely condescending, but I think interpreting guinea-unicorn's post that way doesn't make sense. Even in your reply you didn't say you think it's the right interpretation, just that someone might believe the NSA could "only be up to evil". That followup gives the impression you were giving an FYI for readers. Which is nice to do, but then the "what about" doesn't fit.
So all of that is to say the words "what about" felt like you were deciding to read their post in an unfair way.
I'm happy to listen to an alternate explanation! But you ignored my request for why you said that, and I'm honestly kind of confused as to why that's what set you off.
So overall I think I think my first post can come across as fighty but I don't think the followups should suggest I'm making things fighty. I think my response to 2OEH8eoCRo0 was fine given the way they were ignoring half of the four sentences I had typed.
Notably that was a "no-one-but-us" backdoor, that requires a specific secret key to exploit. We'll see when someone analyzes the payload further, but presumably this backdoor also triggers on a specific private key. If not there are ways to do it that would look far more like an innocent mistake, like a logic bug or failed bounds check.
I can see some arguments that might persuade the NSA to run an attack like this
- gathers real world data on detection of supply attacks
- serves as a wake-up call for a software community that has grown complacent on the security impact of dependencies
- in the worst case, if no one finds it then hey, free backdoor
You say that as if members of US government agencies didn't plot terror attacks on Americans (Operation Northwood), steal the medical records of American whistleblowers (Ellsberg), had to be prevented from assassinating American journalists (Gordon Liddy, on Jack Anderson), collude to assassinate American political activists (Fred Hampton), spy on presidential candidates (Watergate), sell weapons to countries who'd allegedly supported groups who'd launched suicide bombing attacks on American soldiers (Iran-Contra), allow drug smugglers to flood the USA with cocaine so that they could supply illegal guns to terrorists abroad on their return trip (Iran-Contra again) and get caught conducting illegal mass-surveillance on American people as a whole (Snowden). Among others.
It's super-naive to suggest that government agencies wouldn't act against the interest of American citizens and companies because there might be consequences if they were caught. Most of the instances above actually were instances where the perpetrators did get caught, which is why we know about them.
You don’t even have to be this conspiratorially minded to believe the NSA is a legitimate suspect here. (For the record, I think literally every intelligence agency on Earth is plausible here.)
You kind of lost the thread when you say, “act against the interests of American citizens and companies”. Bro, literally anyone could be using xz, and anyone could be using Red Hat. You’re only “acting against Americans” if you use it against Americans. I don’t know who was behind this, but a perfectly plausible scenario would be the NSA putting the backdoor in with an ostensibly Chinese login and then activating on machines hosted and controlled by people outside of the US.
Focusing on a specific distro is myopic. Red Hat is popular.
> but a perfectly plausible scenario would be the NSA putting the backdoor in with an ostensibly Chinese login and then activating on machines hosted and controlled by people outside of the US.
There's a term for that: NOBUS (https://en.wikipedia.org/wiki/NOBUS). It won't surprise me at all if this backdoor can only be exploited if the attacker has the private key corresponding to a public key contained in the injected code. It also won't surprise me if this private key ends up being stolen by someone else, and used against its original owner.
The HN crowd has come a long way from practically hero-worshipping Snowden to automatically assuming that 'state actor' must mean the countries marked evil by the US.
I'd say that CCTV is quite different to wiretapping. You (generally) wouldn't have the expectation of privacy in a public place, most people would expect that phone calls, messages, etc do remain private.
Now, GCHQ is no better than the NSA for that either, but I don't think CCTV is a good comparison.
While his leaks expose surveillance, he was useful idiot https://en.wikipedia.org/wiki/Useful_idiot in hands of Assange club. And it might be event of his saving was trigger for Putin to start war. So no, I'd better see whole camaraderie before court and sentenced. Regardless of 'heroism'.
And yes, most of modern supporters of Wikileaks / Assange / Snowden / etc, chanting 'release Assange' and 'pardon Snowden' are useful idiots in hands of tyrannies like BRICS club.
I'm not very inclined to think this is the US govt, however, you should better acquaint yourself with the morals of some members of Congress.
I think the best reason to doubt USG involvement is the ease with which somebody discovered this issue, which is only a month or two old. I feel like NSA etc. knows not to get caught doing this so easily.
Yeah as we know, intelligence agencies are very often held accountable in the US. As witnessed by all the individuals that got charged or punished for uh... nevermind.
Given the details from another comment [1], it sounds like both maintainers are suspicious. Lasse's behavior has changed recently, and he's been pushing to get Jia Tan's changes into the Linux kernel. It's possible both accounts aren't even run by the original Lasse Collin and Jia Tan anymore.
Edit: Also, Github has suspended both accounts. Perhaps they know something we don't.
Whoops, I linked the wrong comment. I meant to link this one [1]. Anyway, seems like there's potentially a whole trail of compromised and fake accounts [2]. Someone in a government agency somewhere is pretty disappointed right now.
According to Webarchive, https://tukaani.org/contact.html changed very recently (between 11/02/2024 and 29/02/2024) to add Lasse Collin's PGP key fingerprint. That timing is weird, considering his git activity at that time is almost non existent. Although, i checked, this key existed back in 2012.
> Generally in the blackhat world, attackers have very precise targets
Lol, what
> wants to cast a wide net to attack as many as possible. So that fits the profile of a government intelligence agency
That's quite backwards. Governments are far more likely to deploy a complex attack against a single target (see also: Stuxnet); other attackers (motivated primarily by money) are far more likely to cast a wide net.
> That's quite backwards. Governments are far more likely to deploy a complex attack against a single target (see also: Stuxnet); other attackers (motivated primarily by money) are far more likely to cast a wide net.
Governments are well known to keep vulnerabilities hidden (see EternalBlue). Intentionally introducing a vulnerability doesn’t seem that backwards tbh
Oh for sure. I'm not suggesting that this wasn't a government actor, although I'd only give you 50/50 odds on it myself. It coulda just been someone with a bunch of time, like phreakers of old.
Adding some unreadable binary to the source code is a really dangerous thing to do.
We also need tools to quickly detect the addition of indentation symbols that can be easily overlooked.
BYW,I had a classmate who used to play DOTA1(on war3) under this name at the University of Science and Technology of China a long time ago, and this was his first girlfriend name (maybe) . His father was a high-ranking official. Then he joined the parent department of the Internal Security Detachment, a secret service that has gained a lot of power in the last few years. I hope I'm not awake . lol.
Seems to be a perfect project to hijack. Not too much happening, widely used, long history, single maintainer who no longer has time to manage the project and wants to pass it over.
Not a developer but reading the changelogs and commit history from this person seem interesting, as they appear to be some effort consolidate control and push things in the direction of supporting wider dissemination of their backdoor code:
Discussing commits that the other author has since reverted, IFUNC change with Project Zero tests, a focus on embedded, etc.:
"crazytan" is the LinkedIn profile of a security software engineer named Jia Tan in Sunnyvale working at Snowflake, who attended Shanghai Jiao Tong University from 2011 to 2015 and Georgia Institute of Technology from 2015 to 2017. However, this Jia Tan on LinkedIn might not be the same Jia Tan who worked on XZ Utils. Also, the person who inserted the malicious code might be someone else who hijacked the account of the Jia Tan who worked on XZ Utils.
He understood the software architecture quite early on while working on the following repository. He connected the dots from his other projects and went rogue. (probably to benefit from crypto?). Take a look at his other repositories and code style and recent likes on github. Is he our Jia Tan?
My assumption would be that he knows the jig is up, and is probably going to do everything he can to jettison the JiaTan account, lest any IPs he uses be turned over to authorities.
An Indian with the name, Jigar (meaning heart) would never address himself as Jigar, as seen in the citation. This would be culturally a bit weird. Unless he is being sarcastic or writing this on some comic note.
Secondly, the use of English is not consistent in what should be from typical Indian. He should be from a foreign background or a very reputed English medium.
The language though seemingly simple for a native English speaker but it seems in this case; a person whose first language: likely is not English.
It is possible that Grammarly or auto correct could have been used to write these. But can't be certain of anything stated above.
I do think that this is a sabotage account with 60% chances unless Mr. Kumar comes out clean, publicly. He is likely a state sponsored actor.
1,963 comments
[ 3.4 ms ] story [ 672 ms ] threadHowever, I don't like it much and I think software should be compiled for the target machine in the first place. My 1 hardened system that is reachable from the public network is based on musl, built mostly with llvm, and with ifunc disabled.
That means you either have to compile software locally on each machine, or you have a combinatorial explosion of possible features.
Compiling locally has several drawbacks. It needs the full compilation environment installed on every machine, which uses a lot of disk space, and some security people dislike it (because then attackers can also compile software locally on that machine); compiling needs a lot of memory and disk space, and uses a lot of processor time and electric power. It also means that signature schemes which only allow signed code cannot be used (or you need to have the signing key available on the target machine, making it somewhat pointless).
The combinatorial explosion of features has been somewhat tamed lately, by bundling sets of feature into feature levels (x86_64-v1, etc), but that still quadruples the amount of compiled code to be distributed, and newer features still have to be selected at runtime.
Or you just have to buy a lot of the exact same hardware. Secure installations tend to do that.
By "open question" I meant that there is compelling research indicating that GNU memcpy/memcmp is counterproductive, but the general Linux-using public did not get the memo.
https://storage.googleapis.com/gweb-research2023-media/pubto...
"AsmDB: Understanding and Mitigating Front-End Stalls in Warehouse-Scale Computers" Section 4.4 "Memcmp and the perils of micro-optimization"
Wasn't this the point of Gentoo, back in the day? It was more about instruction scheduling and register allocation differences, but your system would be built with everything optimized for your uarch.
FYI this looks for pkgs with liblzma:
> dpkg -l |grep liblzma
Versions >= 5.6 are compromised
The disabling of ifunc in this PR against Google's oss-fuzz project maybe one way they tried to prevent this particular backdoor being flagged by that tool? https://github.com/google/oss-fuzz/pull/10667
https://github.com/llvm/llvm-project/issues/63957
It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?
Ubuntu 22.04 version:
dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library
Whew!
That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...
If I were doing it AND amoral, I'd also be willing to find and compromise committers in various ways.
Apart from (both visible and invisible) taxes, I expect a senior programmer would earn ~500-700k CNY per year. Game programmers may reach up to 200k. For a team able to perform such attack, 1M/yr avg. might be reasonable.
But if this is not a state-sponsored attack, I can't find enough interest. And, if this is state-backed...contractor or some dishonest officials would a huge part, so the real cost might be >2M/yr. Considering you can get nothing during 2 year's lurking I doubt if it's feasible enough.
This is absolutely not the case in many parts of the world.
Given the complexity of the attack, I'd assume the name is fake.
To anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people
Think about what various corps and state-level actors have been putting in there.
The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.
[EDIT: this string gives cleaner results:]
and saw liblzma in the results of both, so there is some sort of similar trickery going on.Anyway. I did not see lzma in the results on Devuan running a process check (just in case). I did see it on a Debian.
It basically is. libsystemd links to liblzma for other features not related to notifications.
(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)
It doesn't matter that libsystemd links to liblzma for other reasons. It's still in the address space of any daemon that is using libsystemd for the notification protocol.
For Slurm, I looked at what a PITA pulling libsystemd into our autoconf tooling would be, stumbled on the Golang implementation, and realized it's trivial to implement directly.
There is AF_UNIX support, but only for streams and not datagrams: https://bugs.openjdk.org/browse/JDK-8297837
What an odd decision. I suppose that you could execute systemd-notify but that's a solution that I would not like.
What I did was to use JNA to call sd_notify() in libsystemd.so.0 (when that library exists), which works but obviously does not avoid using libsystemd. I suppose I could have done all the socket calls into glibc by hand, but doing that single call into libsystemd directly was simpler (and it can be expected to exist whenever systemd is being used).
Unless that feature also has some weird limitation, you could probably use that to call the socket API in libc.
Which is pretty emblematic of systemd's primary architectural fault!
If that split had never happened, then liblzma wouldn't have ended up being linked into sshd...
Note that OP found this in Debian sid as well, which means it's highly unlikely this issue will find its way into any Debian stable systems.
My Arch system was not vulnerable because openssh was not linked to xz.
IMO every single commit from JiaT75 should be reviewed and maybe even rolled back, as they have obliterated their trust.
edit:
https://github.com/google/oss-fuzz/pull/10667
Even this might be nefarious.
Have you come across an outline or graph of systemd that you really like, or maybe a good example of a minimal setup?
Also, only users on sid (unstable) and maybe testing seem to have been affected. I doubt there are many Debian servers out there running sid.
Debian stable (bookworm) has xz-utils version 5.4.1: https://packages.debian.org/bookworm/xz-utils
Guess who released 5.4.1? JiaT75!
https://salsa.debian.org/debian/xz-utils/-/tree/v5.4.1/m4
The script was not present in the git tree, only in the released archives.
I'm also suggesting that there could be more than one exploit present. All of their commits should be rolled back, none of it can be trusted.
I confess I couldn't quite figure out the branching and tagging strategy on that repo. Very weird stuff. That script seems to have been added by Sebastian Andrzej Siewior just ahead of the 5.6.0 release. It's definitely present in the Debian git tree, and probably in many other distros since others seem to be affected.
The commit where the script was added to Debian is tagged `upstream/v5.6.0` despite the script itself not being present on that tag upstream: https://github.com/tukaani-project/xz/tree/v5.6.0/m4
> I'm also suggesting that there could be more than one exploit present. All of their commits should be rolled back, none of it can be trusted.
I agree.
It's just a regular Debian packaging repository, which includes imports of upstream tarballs - nothing out of ordinary there. Debian packaging is based on tarballs, not on git repos (although in absence of upstream tarballs, Debian maintainer may create a tarball out of VCS repo themselves).
The linked repo just happens to include some tags from upstream repo, but those tags are irrelevant to the packaging. Only "debian/*" and "upstream/*" tags are relevant. Upstream VCS history is only imported for the convenience of the packager, it doesn't have to be there.
Debian's git repositories don't have any forced layout (they don't even have to exist or be up-to-date, the Debian Archive is the only source of truth - note how this repo doesn't contain the latest version of the package), but in practice most of them follow the conventions of DEP-14 implemented by gbp (in this particular case, it looks like `gbp import-orig --upstream-vcs-tag`: https://wiki.debian.org/PackagingWithGit#Upstream_import_met...).
Monocrops are more vulnerable to disease because the same (biological) exploit works on the entire population. In our Linux biosphere where there are dozens of major, varied configurations sharing parts but not all of their code (and hundreds or thousands of minor variations), a given exploit is likely to fail somewhere, and that failure is likely to create a bug that someone can notice.
It's not foolproof, but it helps keep the ecosystem healthy.
> These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.
The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:
Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.
What's systemd? 2M lines? It was >1M lines 4+ years ago.
For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.
And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.
Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.
But not yet. So... well, oh well.
The problem? It's on the backburner because I don't think I could find a business model to make money from it.
I don't think offering support for a price would work, for example.
If I am considered a full vendor, though, and a vendor for a critical piece of software, they might keep me around.
If you want you can just use systemd as PID1 for service management and enjoy a sane way to define and manage services – and do everything in archaic ways like 20 years ago.
* Choice. If I have a separate implementation, my users do not have to be subject to systemd's choices. And I do not either.
* The same implementation will have the same bugs, so in the same way that redundant software has multiple independent implementations, having an independent implementation will avoid the same bugs. It may have different bugs, sure, but my goal would be to test like SQLite and achieve DO-178C certification. Or as close as I could, anyway.
* https://jdebp.uk/Softwares/nosh/guide/converting-systemd-uni...
Most of the things you named there are modular and can be easily disabled.
Furthermore, udev precedes systemd and systemd has in fact its own replacement for it (though the name escapes me).
Kind of a classic, people loving harping on systemd without properly understanding it.
That's a common defense for any bloatware. If they're modular and easily disabled then why are they all enabled by default?
I understand that people don't like the way it seems to work itself into the rest of Linux user space as a dependency but that's actually our own fault for not investing the man power that Red Hat invests. We have better things to do than make our own Linux user space and so they have occupied that niche. It's free software though, we always have the freedom to do whatever we want.
By the way, all the stuff you mentioned is not really part of the actual init system, namely PID 1. There's an actual service manager for example and it's entirely separate from init. It manages services really well too, it's measurably better than all that "portable" nonsense just by virtue of using cgroups to manage processes which means it can actually supervise poorly written double forking daemons.
Also, the more extensive the remit (of this init), the more complexly interconnected the interactions between the components; the fewer people understand the architecture, the fewer people understand the code, the fewer people read the code. This creates a situation where the codebase is getting larger and larger at a rate faster than the growth of the number of man-hours being put into reading it.
This has to make it easier for people who are systemd specialists to put in (intentionally or unintentionally) backdoors and exploitable bugs that will last for years.
People keep defending systemd by talking about its UI and its features, but that completely misses the point. If systemd were replaced by something comprehensible and less internally codependent, even if the systemd UI and features were preserved, most systemd complainers would be over the moon with happiness. Red Hat invests too much into completely replacing linux subsystems, they should take a break. Maybe fix the bugs in MATE.
This is a bit of a rich criticism of systemd, given the init scripts it replaced.
> Red Hat invests too much into completely replacing linux subsystems, they should take a break. Maybe fix the bugs in MATE.
MATE isn't a Red Hat project. And nobody complains about Pipewire.
Tell us you never bothered to understand how init worked before drawing a conclusion on it without telling us.
The kernel devs are wasting time writing one offs for every vendor known to man, and it ships to desktops too.
They should do whatever they feel is best for them, as should we. They're releasing free as in freedom GPL Linux software, high quality software at that. Thus I have no moral objections to their activities.
You have to realize that this is really a symptom of others not putting in the required time and effort to produce a better alternative. I know because I reinvent things regularly just because I enjoy it. People underestimate by many orders of magnitude the effort required to make something like this.
So I'm really thankful that I got systemd, despite many valid criticisms. It's a pretty good system, and it's not proprietary nonsense. I've learned to appreciate it.
Init just a more or less normal program that Linux starts by default and by convention. You can make it boot straight into bash if you want. I created a little programming language with the ultimate goal of booting Linux directly into it and bringing up the entire system from inside it.
It's just a normal process really. Two special cases that I can think of: no default signal handling, and it can't ever exit. Init will not get interrupted by signals unless it explicitly configures the signal dispositions, even SIGKILL will not kill it. Linux will panic if PID 1 ever exits so it can't do that.
Traditionally, it's also the orphaned child process reaper. Process descriptors and their IDs hang around in memory until something calls wait on them. Parent processes are supposed to do that but if they don't it's up to init to do it. Well, that's the way it works traditionally on Unix. On Linux though that's customizable with prctl and PR_SET_CHILD_SUBREAPER so you actually can factor that out to a separate process. As far as I know, systemd does just that, making it more modular and straight up better than traditional Unix, simply because this separate process won't make Linux panic if it ever crashes.
As for the service manager, this page explains process and service management extremely well:
https://mywiki.wooledge.org/ProcessManagement
Systemd does it right. It does everything that's described in there, does it correctly, uses powerful Linux features like cgroups for even better process management and also solves the double forking problem described in there. It's essentially a solved problem with systemd. Even the people who hate it love the unit files it uses and for good reason.
The thing that people usually complain about is systemd forcibly setting its process manager at pid=1. I.e. the thing "discussed" in https://github.com/systemd/systemd/issues/12843
There is a secondary feature to run per-user managers, though I'm unsure whether it does run doesn't run without systemd PID1. Though it might only rely on logind.
Makes no sense to me why the service manager part would require running as PID 1. The maintainer just says this:
> PID 1 is very different from other processes, and we rely on that.
He doesn't really elaborate on the matter though.
Every time this topic comes up I end up searching for those so called PID 1 differences. I come up short every time aside from the two things I mentioned above. Is this information buried deep somewhere?
Just asked ChatGPT about PID 1 differences. It gave me the aforementioned two differences, completely dismissed Linux's prctl child subreaper feature "because PID 1 often assumes this role in practice" as well as some total bullshit about process group leaders and regular processes not being special enough to interact with the kernel which is just absolute nonsense.
So I really have no idea what it is about PID 1 that systemd is supposedly relying on that makes it impossible to split off the service manager from it. Everything I have read up until now suggests that it is not required, especially on Linux where you have even more control and it's not like systemd is shy about using Linux exclusive features.
Except it literally is. I once had a systemd system suddenly refuse to boot (kernel panic because PID1 crashed or so) after a Debian upgrade, which I was able to resolve by... wait for it... making /etc/localtime not be a symlink.
Why does a failure doing something with the timezone make you unable to boot your system? What is it even doing with the timezone? What is failing about it? Who knows, good luck strace'ing PID1!
I was also corrected further down in the thread, with citations from the maintainers even:
https://news.ycombinator.com/item?id=39871735
As it stands I really have no idea why the service manager has not been split off from PID 1. Maintainer said that PID 1 was "different" but didn't really elaborate. Can't find much reliable information about said differences either. Do you know?
That's my entire problem with systemd though: despite the averred modularity, it combines far too many concerns for anyone to understand how or why it works the way it does.
The other difference is PID 1 can't exit because Linux panics if it does. That's actually an argument for moving functionality out of PID 1.
There are other service managers out there which work outside PID 1. Systemd itself literally spawns non-PID 1 instances of itself to handle the user services. I suppose only the maintainers can tell us why they did it that way.
Maybe they are relying on the fact PID 1 traditionally reaps zombies even though Linux has a prctl for that:
https://www.man7.org/linux/man-pages/man2/prctl.2.html
What if the issue is just that nobody's bothered to write the code to move the zombie process reaping to a separate process yet? Would they accept patches in that case?Ludicrously, that manual page straight up says systemd uses this system call to set itself up as the reaper of zombie processes:
> Some init(1) frameworks (e.g., systemd(1)) employ a subreaper process
If that's true then I really have no idea what the hell it is about PID 1 that they're relying on.
Edit: just checked the source code and it's actually true.
https://github.com/systemd/systemd/blob/main/src/core/main.c...
https://github.com/systemd/systemd/blob/main/src/basic/proce...
https://github.com/systemd/systemd/blob/main/src/basic/proce...
So they're not relying on the special signals handling and they even have special support for non-PID 1 child subreapers. Makes no sense to me. Why can't they just drop those PID == 1 checks and make a simpler PID 1 program that just spawns the real systemd service manager?
Edit: they already have a simple PID 1 in the code base!
https://github.com/systemd/systemd/blob/main/src/nspawn/nspa...
It's only being used inside namespaces though! Why? No idea.
I actually kinda think that can be an advantage for a service manager. If your service manager crashes an automatic reboot is nice, in a way. I doubt that's why they did it though.
I don't think it's gonna do that! I saw it in the source code: when it's running as PID 1, systemd installs a crash handler that freezes itself in a desperate attempt to avoid the kernel panic! It's pretty amazing. They could have written it so that PID 1 watches over the service manager and just restarts it if it ever crashes. I mean, systemd already supports soft-rebooting the entire user space which is pretty much exactly what would happen if PID 1 restarted a separate service manager.
Know what else I found in the source code? Various references to /proc/1. I'm starting to think that's the true reason why they want to be PID 1...
systemd is a collection of tools, one of which is an init system. Nobody accused GNU yes of being bloated just because it's in a repository alongside 50 other tools.
[1] https://github.com/coreutils/coreutils/blob/master/src/yes.c
But that's the very problem with systemd! As time goes on you're required, whether by systemd itself or by the ecosystem around it, to use more and more of it, until it's doing not only service management but also timezones, RTC, DNS resolution, providing getpwent/getgrent, inetd, VMs and containers, bootloader, udev (without adding literally any benefit over the existing implementations), ... oh and you also have to add significant complexity in other things (like the kernel!) to use it, like namespaces (which have been a frequent source of vulnerabilities)...
How many of those are you actually required to use systemd for? At least for DNS, inetd, containers and bootloader I'm pretty sure I run a few different alternatives across my systems. I think major distros (running systemd) still ship with different dns and inetd, for containers its a lot more common to use a docker-like (probably docker or podman) than it is to use systemd-nspawn.
> oh and you also have to add significant complexity in other things (like the kernel!) to use it, like namespaces (which have been a frequent source of vulnerabilities)
Namespaces were implemented before systemd, have been used before systemd in widely used systems (for example LXC and many others). Namespaces and similar kernel features are not tied to systemd.
That depends on what other software you want to run, because systemd's design heavily encourages other things (distros, libraries, applications) to take dependencies on various bits. See also: every mainstream distro.
> Namespaces were implemented before systemd, have been used before systemd in widely used systems (for example LXC and many others). Namespaces and similar kernel features are not tied to systemd.
Didn't say they were. But I don't have to use LXC or many others in order to use the most popular distros and applications.
I do have to use systemd for that, though.
Which means I have to have namespaces enabled.
Unprivileged user namespaces sure, but I don't think that applies to namespaces in general (which without unprivileged user namespaces can only be created by root, and LPE is the concern with unprivileged userns due to increased attack surface). systemd doesn't need unprivileged userns to run.
this is and always has been such a dumb take.
if you'd like to implement an init (and friends) system that doesn't have "unnecessary complexity" and still provides all the functionality that people currently want, then go and do so and show us? otherwise it's just whinging about things not being like the terrible old days of init being a mass of buggy and racey shell scripts.
Zero of the major distros used System V init by default. Probably only distros like Slackware or Linux From Scratch even suggested it.
It's unfortunate that so many folks uncritically swallowed the Systemd Cabal's claims about how they were the first to do this, that, or the other.
(It's also darkly amusing to note that every service that has nontrivial pre-start or post-start configuration and/or verification requirements ends up using systemd to run at least one shell script... which is what would have often been inlined into their init script in other init systems.)
I have absolutely no idea what you're trying to claim.
Are you suggesting that Debian's "sysvinit" package wasn't a System V init system? That the years I spent editing shell scripts in /etc/init.d/ wasn't System V init?
or are you making some pointless distinction about it not actually being pre-lawsuit AT&T files so it doesn't count or something?
or did you not use Linux before 2010?
if you have some important point to make, please make it more clearly.
> It's unfortunate that so many folks uncritically swallowed the Systemd Cabal's claims about how they were the first to do this, that, or the other.
I feel like you have very strong emotions about init systems that have nothing to do with the comment you're replying to.
I've been using Linux regularly since 2002. I've never regularly used a Linux that used sysvinit.
In other words, over the past ~22 years (goddamn, where did the time go?) every Linux I've regularly used has had an init system that allows you to specify service dependencies to determine their start order.
> ...Debian...
Ah. That explains it. Debian's fine to build on top of but a bad distro to actually use. (Unless you really like using five-to-ten (and in some cases 25->35) year old software that's been superseded by much-improved versions.)
You should also consider that packages named "sysvinit" sometimes aren't actually what people think of when they hear "sysvinit": <https://wiki.gentoo.org/wiki/Sysvinit>
Complexity that would otherwise be distributed to a sea of ad-hoc shell scripts? systemd is a win
They are never "simple", there is always some fucking edge case, like for example we had Java apps writing its own PID few seconds after start. So any app that did start and immediately after status (like Pacemaker) threw errors.
Or how once in blue moon MySQL didn't start after reboot because it so happened that:
* the PID file from previous boot wasn't cleared * some other app ran with same PID as it was in file * Script did not care, script saw pid file existing and didn't start MySQL.
Both examples from pre-systemd CentOS
Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?
2. You can run Debian systems without systemd as PID 1, but you're still stuck with libsystemd because so many daemons now link with it.
Socket activation and the NFS automounter appear to.
If I run "netstat -ap" I see pid 1 listening on enabled units.
Edit: tinysshd is specifically launched this way.
Edit2: there is also substantial criticism of xz on technical grounds.
https://www.nongnu.org/lzip/xz_inadequate.html
But let me stress two other things:
Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.
With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.
(I really wish there were a way to link such that the library isn't actually loaded but it still shows in the metadata, so you can get the performance benefits of doing less work but can still analyze the dependency DAG easily)
Dlopen has drawbacks but also major benefits. We decided the benefits relatively clearly outweigh the drawbacks, but of course people may disagree.
I have proposed a mechanism before, that would expose the list of libs we potentially load via dlopen into an ELF section or ELF note. This could be consumed by things such as packagae managers (for auto-dep generation) and ldd. However there was no interest in getting this landed from anyone else, so I dropped it.
Note that there are various cases where people use dlopen not on hardcoded lib names, but dynamically configured ones, where this would not help. I.e. things like glibc nss or pam or anything else plugin based. But in particular pam kinda matters since that tends to be loaded into almost any kind of security relavant software, including sshd.
This might sound like a lot of work for a package-manager-less-language ecosystem at first, but if you consider "tag" as "exports symbol with name", it is in fact already how most C plugin systems work (a few use an incompatible per-library computed name though, or rely entirely on global constructors). So really only the loading programs need to be modified, just like the fixed-name `dlopen`.
That's what I think too. Do the relevant docs point this out too? Ages ago they didn't. I think we should try to avoid that people just google "implement systemd notify daemon" and end up on a page that says "link to libsystemd and call sd_notify()".
Could you point out where the man page (https://www.freedesktop.org/software/systemd/man/latest/sd_n...) says this?
:-D
Inaccurate.
It's not pulled in on any sysvinit Debian system I run. It is on stable, oldstable, and oldoldstable systems via systemd.
Not systemd:
# ldd $(which sshd) linux-vdso.so.1 (0x00007ffcb57f5000)
systemd:# ldd $(which sshd) linux-vdso.so.1 (0x00007ffc4d3eb000)
I've been learning NixOS for a few years now, and it would have been impossible without systemd. It's one heck of a learning curve, but when you get to the other side, you know something of great power and value. Certain kinds of complexity adds 'land' (eg. systemd) that can become 'real estate' (eg. NixOS), which in turn hopes to become 'land' for the next innovation, and so forth.
Whether this happens or not (whether it's the right kind of complexity) is really hard to assess up-front, and probably impossible without knowing the complex new technology in question very well. (And by then you have the bias of depending, in part, yourself on the success of the new tech, as you've committed significant resources to mastering it, so good luck on convincing skeptical newcomers!)
It's almost like a sort of event horizon -- once you know a complex new technology well enough to see whether or not it's useful, the conflict-of-interest makes your opinion unreliable to outsiders!
Nevertheless, the assessment process itself, while difficult to get right, is worth getting better at.
It's easy for impatience and the sensation of what I've taken to calling 'daunt' -- that intrinsic recoil that the mind has from absorbing a large amounts of information whose use case is not immediately relevant -- to dissuade one from exploring. But then, one never discovers new 'land', and one never builds new real estate!
[ Aside: This is why I'm a little skeptical of the current rebellion against frontend frameworks. Certainly some of them, like tailwind, are clearly adding fetters to an otherwise powerful browser stack. But others, like Svelte, and to some extent, even React, bring significant benefits.
The rebellion has this vibe like, well, users _should_ prefer more simply-built interfaces, and if they don't, well, they just have bad taste. What would be more humble would be to let the marketplace (e.g. consumers) decide what is preferable, and then build that. ]
Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.
Practically speaking it can't - For one the script injected into the build process tests that you're running on x86-64 linux, for another, the injected code is elf code, which wouldn't link on a mac. It also needs to manipulate dynamic linker datastructures, which would also not work the same on a mac.
> This could perhaps affect more than just sshd on x86-64 linux?
This however is true - /usr/sbin/sshd was the only argv[0] value that I found to "work", but it's possible there are others. "/usr/sbin/sshd" isn't a string directly visible in the injected code, so it's hard to tell.
- linux
- x86-64
- building with gcc & the GNU linker
- part of a .deb or .rpm build
Add to that, as the article explains: openssh does not directly use liblzma, the only reason SSH is affected at all, is because some Linux Distros patch openssh to link it against systemd, which does depend on liblzma.
Could it affect things other than SSH on a Mac? Unlikely. The compromise was introduced in 5.6.0, but macOS Sonoma has 5.4.4 (from August last year).
[1]: https://github.com/Homebrew/homebrew-core/pull/167512
Can we even be sure no such successful attempt has already been made?
Remember, there is no such thing as computer security. Make your decisions accordingly :)
I would go further than that: all files which are in a distributed tarball, but not on the corresponding git repository, should be treated as suspect.
Distributing these generated autotools files is a relic of times when it could not be expected that the target machine would have all the necessary development environment pieces. Nowadays, we should be able to assume that whoever wants to compile the code can also run autoconf/automake/etc to generate the build scripts from their sources.
And other than the autotools output, and perhaps a couple of other tarball build artifacts (like cargo simplifying the Cargo.toml file), there should be no difference between what is distributed and what is on the repository. I recall reading about some project to find the corresponding commit for all Rust crates and compare it with the published crate, though I can't find it right now; I don't know whether there's something similar being done for other ecosystems.
Also chances are you can shove that flag in some old `configure.in` and have it work with an old autoconf for years before it having to update it :-P.
Yes, it sucks to add yet another wrapper but that’s what you get for choosing non backwards compatible tools in the first place. In combination with projects that don’t keep up to date on supporting later versions.
A git hash means nothing without the repository it came from, so you'd need to distribute both. A tarball is a self-contained artifact. If I store a tarball in a CD-ROM, and look at it twenty years later, it will still have the same complete code; if I store a git hash in a CD-ROM, without storing a copy of the repository together with it, twenty years later there's a good chance that the repository is no longer available.
We could distribute the git hash together with a shallow copy of the repository (we don't actually need the history as long as the commit with its trees and blobs is there), but that's just reinventing a tarball with more steps.
(Setting aside that currently git hashes use SHA-1, which is not considered strong enough.)
And git even has support for "compressed git repo in a file" or "shallow git repo in a file" or even "diffs from the last release, compressed in a file". They're called "git bundle"'s.
They're literally perfect for software distribution and archiving.
People also like version numbers like 2.5.1 but that's not a hash, and you can only indirectly make it a hash.
This and the automated A/B / diff to check the tarball against the repo, flag if mismatched.
How long do we need to (pretend to) keep compatibility with pre-ANSI C compilers, broken shells on exotic retro-unixes, and running scripts that check how many bits are in a byte?
For fun, you can read the responses to my musing that maybe build systems aren't needed: https://news.ycombinator.com/item?id=35474996 (People can't imagine programming without a build system - it's sad)
Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?
Crazy indeed.
Autoconf is bad in this respect but it's not like the alternatives are better (maybe Bazel).
If I were to sneak in some underhanded code, I'd do it through either a dependency that is used by build.rs (not unlike what was done for xz) or a crate purporting to implement a very useful procedural macro...
Some things are just that complex.
You figure out what the hardware designers actually did, and get the program written to accommodate it.
Sure - you want to test stuff, but that can be done with a special "test build" in it's own VM.
This incident shows that killing the autoconf goop is long overdue.
Mea culpa!
But that's not really important to the point - I'm not looking at a diff of every committed favicon.ico or ttf font or a binary test file to make sure it doesn't contain a shellcode.
A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".
Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".
[0] https://news.ycombinator.com/item?id=39154825
In (pre-backdoor) xz 5.4.5:
unfortunately thats just standard in configure scripts, for example from python:
``` grep eval Python-3.12.2/configure | wc -l 165 ```
and its 32,958 lines of code, plenty of binary fixtures as well in the tarball to hide stuff.
who knows, but I have feeling us finding the backdoor in this case was more of a happy accident.
It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.
Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)
> b) argv[0] needs to be /usr/sbin/sshd
For once, the lack of FHS interoperability is a benefit, if only on accident.
Does that mean this affects RHEL and Fedora?
https://www.redhat.com/en/blog/urgent-security-alert-fedora-...
https://lists.debian.org/debian-security-announce/2024/msg00...
A supply chain attack serve as the basis for another supply chain attack.
and when they do port finally backport this bug in 2026, they will probably implement the systemd integration with openssl (pbthththt...) via 600 patch files in some nonstandard divergent manner that thwarts the payload anyhow. see? i knew they were super duper secure.
I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.
That is actually a major point of a lot of corporate security measures (shifting risk)
5.6.1 was available for a few days and just rolled back ~20 minutes ago: https://github.com/macports/macports-ports/commit/a1388aee09...
I was going to uninstall but it's used by so many things…
Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.
Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).
Certification theater.
It's completely performative.
And that's a good thing.
I agree on the sshd linking part.
Note that I'm not looking for the vulnerable symbols, I'm looking for the library that does the patching in the first place.
(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)
I've never had to do it myself but I believe that's common practice with embargos on security vulnerabilities.
Also, https://mastodon.social/@mgorny@treehouse.systems/1121802382... from a Gentoo dev mentions that Gentoo doesn't use the patch that results in sshd getting linked against liblzma.
As far as I know this is not an official communication channel so don't take it as such.
But with pacman/makepkg 6.1 (which recently released) git sources can also now be check summed IIRC which is a funny coincidence.
https://archlinux.org/news/the-xz-package-has-been-backdoore...
Makes you wonder what more competent actors can do.
1. Parse command line arguments
2. Setup logging
3. Load configuration files
4. Load keys/certificates into memory (notably including private keys)
5. Listen on a socket/port for incoming connections
6. Spawn a child process with reduced permissions (on Linux, using seccomp filters [2]) to respond to each incoming connection request
This backdoor executes at order 0 before sshd's main function is invoked, overwriting internal sshd functions with compromised ones. As some ideas of what the backdoor could achieve:
1. Leak server private keys during handshakes with users (including unauthenticated users) allowing the keys to be passively stolen
2. Accept backdoor keys as legitimate credentials
3. Compromise random number generation to disable perfect forward secrecy
4. Execute code on the host (supplied remotely by a malicious user) with the 'root' permissions available to sshd upon launch. On most Linux distributions, systemd-analyze security sshd.service will give a woeful score of 9.6/10 (10 being the worst).[3] There is essentially NO sandboxing used because an assumption is made that you'd want to login as root with sshd (or sudo/su to root) and thus would not want to be restricted in what filesystem paths and system calls your remote shell can then invoke.
The same attacker has also added code to Linux kernel build scripts which causes xz to be executed (xz at this point has a backdoor compiled into it) during the build of the Linux kernel where xz compression is used for the resulting image. Using this approach, the attacker can selectively choose to modify certain (or all) Linux kernel builds to do some very nasty things:
1. Leak Wireguard keys allowing them to be passively intercepted.
2. Compromise random number generation, meaning keys may be generated with minimal entropy (see Debian certificate problem from a few years ago).
3. Write LUKS master keys (keys used by dm-crypt for actually decrypting disks) to disks in retrievable format.
4, Introduce remote root code execution vulnerabilities into basic networking features such as TCP/IP code paths.
[1] 'main' function: https://anongit.mindrot.org/openssh.git/tree/sshd.c
[2] https://anongit.mindrot.org/openssh.git/tree/sandbox-seccomp...
[3] https://github.com/gentoo/gentoo/blob/HEAD/net-misc/openssh/...
The way they went around it, however, was brilliant. Completely reduce the variables to directly target whatever it is you're attacking. Reminds me of stuxnet somewhat.
reminds me of the gnu hack discovered because one of the savannah build hosts was some odd architecture the exploit wasn't expecting
Also, we only catch the ones that we ... catch. The ones that do everything perfectly, unless they come out and confess eventually, we don't get to "praise" them for their impeccable work.
He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
1. His machine was compromised, he wasn't at fault past having less than ideal security (a sin we are all guilty of). His country or origin/residence is of no importance and doxing him isn't fair to him.
2. This account was malicious. There's no reason we should believe that the identity behind wasn't fabricated. The country of origin/residence is likely falsified.
In neither case is trying to investigate who he is on a public forum likely to be productive. In both cases there's risk of aiming an internet mob at some innocent person who was 'set up'.
Anyways, yes it is an interesting question whether he/she is alone or they are a group. Conway's law probably applies here as well. And my hunch in general is that these criminal mad minds operate individually / alone. Maybe they are hired by an agency but I don't count that as a group effort.
1. You don't actually know what has been done by whom or why. You don't know if the author intended all of this, or if their account was compromised. You don't know if someone is pretending to be someone else. You don't know if this person was being blackmailed, forced against their will, etc. You don't really know much of anything, except a backdoor was introduced by somebody.
2. Assuming the author did do something maliciously, relying on personal reputation is bad security practice. The majority of successful security attacks come from insiders. You have to trust insiders, because someone has to get work done, and you don't know who's an insider attacker until they are found out. It's therefore a best security practice to limit access, provide audit logs, sign artifacts, etc, so you can trace back where an incursion happened, identify poisoned artifacts, remove them, etc. Just saying "let's ostracize Phil and hope this never happens again" doesn't work.
3. A lot of today's famous and important security researchers were, at one time or another, absolute dirtbags who did bad things. Human beings are fallible. But human beings can also grow and change. Nobody wants to listen to reason or compassion when their blood is up, so nobody wants to hear this right now. But that's why it needs to be said now. If someone is found guilty beyond a reasonable doubt (that's really the important part...), then name and shame, sure, shame can work wonders. But at some point people need to be given another chance.
If it were me I'd be doing damage control to clear my name if my account was hacked and abused in this manner.
Otherwise if I was doing this knowing full well what would happen then full, complete defederation of me and my ability to contribute to anything ever again should commence -- the open source world is too open to such attacks where things are developed by people who assume good faith actors.
Mitnick reformed after he was convicted (whether you think that was warranted or not). Here if these folks are Mitnick’s or bad actors etc let’s get all the facts on the table and figure this out.
What’s clear is that we all need to be ever vigilant: that seemingly innocent patch could be part of a more nefarious thing.
We’ve seen it before with that university sending patches to the kernel to “test” how well the core team was at security and how well that went over.
Anyways. Yeah. Glad you all allowed me to grow. And I learned that I have an emotional connection to open source for better or worse: so much of my life professional and otherwise is enabled by it and so threats to it I guess I take personally.
There is no requirement to use your real name when contributing to open source projects. The name of the backdoor author ("Jia Tan") might be fake. If it isn't, and if somehow they are found to be innocent (which I doubt, looking at the evidence throughout the thread), they can create a new account with a new fake identity.
I bet they intended for their back door to eventually be merged into the base Amazon Linux image.
So the really strange thing is why they put so little effort into making this undetectable. All they needed was to make it use less time to check each login attempt.
It just requires the SSH port to be reachable unless there is also a callout function (which is risky as people might see the traffic). And with Debian and Fedora covered and the change eventually making its way into Ubuntu and RHEL pretty much everything would have this backdoor.
> Thanks to Hans Jansen for the original patch.
https://github.com/tukaani-project/xz/pull/53
There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.
The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...
Also seeing this bug. Extra valgrind output causes some failed tests for me. Looks like the new version will resolve it. Would like this new version so I can continue work.
--
Wow.
(Edited for clarity.)
for example, https://github.com/google/oss-fuzz/pull/10667
From https://salsa.debian.org/users/hjansen/activity
Author: Hans Jansen <hansjansen162@outlook.com>
- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!2 New upstream version 1.17" - March 17, 2024
- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!1 Update to upstream 1.17" - March 17, 2024
- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!2 New upstream version 1.17.0" - March 17, 2024
- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!1 Update to upstream 1.17.0" - March 17, 2024
- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!6 Update upstream branch to 0.10.6" - March 17, 2024
- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!5 Update to upstream 0.10.6" - March 17, 2024
- [Debian / Xz Utils](https://salsa.debian.org/debian/xz-utils): opened merge request "!1 Update to upstream 5.6.1" - March 17, 2024
An internal call via ifunc is not magic — it’s just a call via the GOT or PLT, which boils down to function pointers. An internal call through a hidden visibility function pointer (the right way to do this) is also a function pointer.
The even better solution is a plain old if statement, which implements the very very fancy “devirtualization” optimization, and the result will be effectively predicted on most CPUs and is not subject to the whole pile of issue that retpolines are needed to work around.
Are they really two people conspiring?
Unless proven otherwise, it is safe to assume one is just a pseudonym alias of the other.
Jia Tan getting maintainer access looks like it is almost certainly to be part of the operation. Lasse Colling mentioned multiple times how Jia has helped off-list and to me it seems like Jia befriended Lasse as well (see how Lasse talks about them in 2023).
Also the pattern of astroturfing dates back to 2022. See for example this thread where Jia, who has helped at this point for a few weeks, posts a patch, and a <name><number>@protonmail (jigarkumar17) user pops up and then bumps the thread three times(!) lamenting the slowness of the project and pushing for Jia to get commit access: https://www.mail-archive.com/xz-devel@tukaani.org/msg00553.h...
Naturally, like in the other instances of this happening, this user only appears once on the internet.
I understand it's unlikely, but is there anything I can do to check if the backdoor was used? Also any other steps I should take after "brew upgrade"?
>> Looks like that Homebrew users (both macOS and Linux, both Intel and ARM) are unlikely affected?
> Correct. Though we do not appear to be affected, this revert was done out of an abundance of caution.
[1] https://github.com/Homebrew/homebrew-core/pull/167512
(or SHAs, etc.)
(EDIT: 5.6.0 and 5.6.1 ?)
(EDIT 2: Ooof, looks like the nix unstable channel uses xz 5.6.1 at this time)
I use Nix to manage this stuff on Mac, not Homebrew...
https://www.nongnu.org/lzip/xz_inadequate.html
[0]: https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.
I have dreamed about an automated LLM system that can "diff" the changes out of the binary and provide some insight. You know give back a tiny bit of power to the user. I'll keep dreaming.
> We tuned up the engine and gave the interiors a thorough clean. Everything is now running smoothly again.
Yeah nah mate, if every release is the first release where everything is running smoothly, I'm not going to believe it this time either.
Makes me wonder if the team has some release quota to fill and will push a build even if nothing meaningful has actually changed.
[1] https://slack.com/release-notes/mac
don't miss out on the quality code, like the line that has: i += 4 - 2;
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=50255feeaab...
That's confirmed
From https://www.openwall.com/lists/oss-security/2024/03/29/4:
> The files containing the bulk of the exploit are in an obfuscated form in
> tests/files/bad-3-corrupt_lzma2.xz
> tests/files/good-large_compressed.lzma
> committed upstream. They were initially added in
> https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf...
I don't know how you can be missing the essence of the problem here or that comments point.
Vague claims are meaningless and valueless and are now even worse than that, they are a red flag.
Please don't tell me that you would accept a pr that didn't explain what it did, and why it did it, and how it did it, with code that actually matched up with the claim, and was all actually something you wanted or agreed was a good change to your project.
Updating to the next version of a library is completely unrelated. When you update a library, you don't know what all the changes were to the library, _but the librarys maintainers do_, and you essentially trust that librarys maintainers to be doing their job not accepting random patches that might do anything.
Updating a dependency and trusting a project to be sane is entirely a different prospect from accepting a pr and just trusting that the submitter only did things that are both well intentioned and well executed.
If you don't get this then I for sure will not be using or trusting your library.
https://github.com/JiaT75 was suspended for a moment, but isn't anymore?
The following page for each other show both accounts suspended indeed.
https://github.com/Larhzu?tab=following
https://github.com/JiaT75?tab=following
Super frustrating when trying to develop automation. :(
Simply showing one giant page saying "This respository is disabled" is not helpful in any way.
E.g.: I have a specific JPEG committed into a repository because it triggers a specific issue when reading its metadata. It's not just _random_ data, but specific bogus data.
But yeah, if the test blob is purely random, then you can just commit a seed and generate in during tests.
EDIT: Lasse Collin's account @Larhzu has also been suspended.
EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.
--
EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn't appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:
https://gist.github.com/Qix-/f1a1b9a933e8847f56103bc14783ab7...
--
Here's a mailing list message from them ca. 2022.
https://listor.tp-sv.se/pipermail/tp-sv_listor.tp-sv.se/2022...
--
MinGW w64 on AUR was last published by Jia on Feb 29: https://aur.archlinux.org/cgit/aur.git/log/?h=mingw-w64-xz (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)
--
pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: https://aur.archlinux.org/cgit/aur.git/?h=pacman-static
EDIT: I've emailed the maintainer to have the key removed.
--
Alpine was patched as of 6 hours ago.
https://git.alpinelinux.org/aports/commit/?id=982d2c6bcbbb57...
--
OpenSUSE is still listing Jia's public key: https://sources.suse.com/SUSE:SLE-15-SP6:GA/xz/576e550c49a36... (cross-ref with https://web.archive.org/web/20240329235153/https://tukaani.o...)
EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.
--
Arch appears to still list Jia as an approved publisher, if I'm understanding this page correctly.
https://gitlab.archlinux.org/archlinux/packaging/packages/xz...
EDIT: Just sent an email to the last committer to bring it to their attention.
EDIT: It's been removed.
--
jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.
/whowas expired not too long ago, unfortunately. If anyone has it I'd love to know.They are not registered on freenode.
EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).
EDIT: Jia Tan was usi...
I respect them for that.
Violating that code is just as bad as the bad actor slipping backdoors.
https://git.alpinelinux.org/aports/log/main/gettext
libunistring could also be affected as that has also been pushed there
No.
> libunistring could also be affected as that has also been pushed there
What do you mean by "that"?
Why? Isn't it better to freeze them and let as many people as possible analyze the code?
1. They don't have the ability to freeze repos (i.e. would require some engineering effort to implement it), as I've never seen them do that before.
2. Many distros (and I assume many enterprises) were still linking to the GitHub releases to source the infected tarballs for building. Disabling the repo prevents that.
The infected tarballs and repo are still available elsewhere for researchers to find, too.
https://git.tukaani.org/
It looks like one of Jia Tan's commits (328c52da8a2) added a stray "." character to a piece of C code that was part of a check for sandboxing support, which I guess would cause the code to fail to compile, causing the check to fail, causing the sandboxing to be disabled.
https://tukaani.org/xz-backdoor/
> Those days are pretty much behind us. Sure, you can compile code and tweak software configurations if you want to--but most of the time, users don't want to. Organizations generally don't want to, they want to rely on certified products that they can vet for their environment and get support for. This is why enterprise open source exists. Users and organizations count on vendors to turn upstreams into coherent downstream products that meet their needs.
> In turn, vendors like Red Hat learn from customer requests and feedback about what features they need and want. That, then, benefits the upstream project in the form of new features and bugfixes, etc., and ultimately finds its way into products and the cycle continues.
"and when the upstream is tainted, everyone drinks poisoned water downstream, simple as that!"
A feature which allowed the exploit to take place, let's put it that way.
Over here: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
> The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.
Multiple suggestions on that thread on how that's a legacy practice that might be outdated, especially in the current climate of cyber threats.
Someone even posted a more thorough gist on what could be done to increase transparency and reduce discrepancies between tarballs and repos: https://gist.github.com/smintrh78/97b5cb4d8332ea4808f25b47c8...
And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity
This is quite common in most (all?) distributions. People are going through lists of outdated packages, updating them, testing them, and pushing them.
My GitHub says exactly who I am!
What would it solve when identity theft happens on a mass scale on a day to day basis?
It'd just ruin the life of some random person whose identity got stolen to create the account…
[0] https://thispersondoesnotexist.com
Just FYI, krygorin4545@proton.me (the latest message before the upload) was created Tue Mar 26 18:30:02 UTC 2024, about an hour earlier than the message was posted.
Proton generates PGP key upon creating the account, with the real datetime of the key (but the key does not include the timezone).
https://youtu.be/IfhMILe8C84
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598
[2]: https://news.ycombinator.com/item?id=39733185
https://github.com/tukaani-project/xz/commit/af071ef7702debe...
Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.
Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.
1. https://www.darpa.mil/program/hybrid-ai-to-protect-integrity...
Easier and easier to hide this junk in amongst them.
This is the sort of case where america's over the top hacking laws make sense.
It would seem unlikely this guy would be also logging into peoples boxes after this.
It seems a much tougher job to link something like this to an intentional unauthorized access.
At this point, we have no confirmed access via compromise.
Do you know of a specific case where the existence of a backdoor has been prosecuted without a compromise?
Who would have standing to bring this case? Anyone with a vulnerable machine? Someone with a known unauthorized access. Other maintainers of the repo?
IANAL but it is unclear that a provable crime has been committed here
Best to leave it at that.
It's not worth your time or the reader's time trying to come up with a technicality to make it perfectly legal to do something we know little about, other than it's extremely dangerous.
Law isn't code, you gotta violate some pretty bedrock principles to pull off something like this and get away with it.
Yes, if you were just a security researcher experimenting on GitHub, it's common sense you should get away with it*, and yes, it's hard to define a logical proof that ensnares this person, and not the researcher.
* and yes, we can come up with another hypothetical where the security researcher shouldn't get away with it. Hypotheticals all the way down.
1. It should be legal to develop or host pen-testing/cracking/fuzzing/security software that can break other software or break into systems. It should be illegal to _use_ the software to gain _unauthorised_ access to others' systems. (e.g. it's legal to create or own lockpicks and use them on your own locks, or locks you've been given permission to pick. It's not legal to gain unauthorised access _using_ lockpicks)
2. It should be illegal to develop malware that _automatically_ gains unauthorised access to systems (trojans, viruses, etc.). However, it should be legal to maintain an archive of malware, limiting access to vetted researchers, so that it can be studied, reverse-engineered and combatted. (e.g. it's illegal to develop or spread a bioweapon, but it's ok for authorised people to maintain samples of a bioweapon in order to provide antidotes or discover what properties it has)
3. What happened today: It should be illegal to intentionally undermine the security of a project by making bad-faith contributions to it that misrepresent what they do... even if you're a security researcher. It could only possibly be allowed done if an agreement was reached in advance with the project leaders to allow such intentional weakness-probing, with a plan to reveal the deception and treachery.
Remember when university researchers tried to find if LKML submissions could be gamed? They didn't tell the Linux kernel maintainers they were doing that. When the Linux kernel maintainers found out, they banned the entire university from making contributions and removed everything they'd done.
https://lkml.org/lkml/2021/4/21/454
https://arstechnica.com/gadgets/2021/04/linux-kernel-team-re...
No, people being polite and avoiding the more direct answer that'd make people feel bad.
The rest of us understand that intuitively, and that it is already the case, so pretending there was some need to work through it, at best, validates a misconception for one individual.
Less important, as it's mere annoyance rather than infohazard: it's wildly off-topic. Legal hypotheticals where a security researcher released "rm -rf *" on GitHub and ended up in legal trouble is 5 steps downfield even in this situation, and it is a completely different situation. Doubly so when everyone has to "IANAL" through the hypotheticals.
This is not unauthorized access, but is also clearly wrong. I'm wondering if its illegal, or if its unauthorized access . . .
Of course. The mere publishing of the exploit is not the criminal part. Its the manner & intent in which it was published that is the problem.
> At this point, we have no confirmed access via compromise.
While i don't know the specifics for this particular law, generally it doesn't matter what you actually did. What is relavent is what you tried to do. Lack of success doesn't make you innocent.
> Who would have standing to bring this case?
The state obviously. This is a criminal matter not a civil one. You don't even need the victim's consent to bring a case.
[IANAL]
See for example page 35 in the Justice Department’s computer crimes handbook (dated, but basically AIUI the same way they still do things) [0]
[0] https://www.justice.gov/d9/criminal-ccips/legacy/2015/01/14/...
Also, I think getting malicious code into a repo counts as a compromise in and of itself.
I would definitely be looking at every single commit though and if it isn't obviously safe I'd be drilling in.
Damage wise, most orgs aren't going to be hurt much by NSA or the Chinese equivalent getting access, but a Nigerian criminal gang? They're far more likely to encrypt all your files and demand a ransom.
For example, change from safe_fprintf to fprintf. It would be appropriate that every commit should be reviewed and either tweaked or re-written to ensure the task is being done in the safest way and doesn't have anything that is "off" or introducing a deviation from the way that codebase standardly goes about tasks within functions.
A lot of eyes are on the code. From all sides. Folks trying to find old unpatched backdoors to exploit or patch.
randomly reverting two years of things across dozens of repositories will break them, almost definitely make them unbuildable, but also make them unreleasable in case any other change needs to happen soon.
all of their code needs to be audited to prove it shouldn't be deleted, of course, but that can't happen in the next ten minutes.
I swear that HN has the least-thought-through hot takes of any media in the world.
The irony is too good.
Slight oversimplification, see https://bugs.debian.org/1068024 discussion.
(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)
And in general, the build system of a large project is doing a lot of work and is considered pretty uninteresting and obscure. Random CMake macros or shell scripts would be just as likely to host bad code.
This is also why I like meson, because it's much more constrained than the others and the build system tends to be more modular and the complex parts split across multiple smaller, mostly independent scripts (written in Python or bash, 20-30 lines max). It's still complex, but I find it easier to organize.
Build systems can even have undefined behaviour in the C++ sense. For example Conan 2 has a whole page on that.
It's entirely possible they could have got that injection through review, even if they had that framwork and instead put it in source files used to generate autoconf soup.
Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563
I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.
Windows started using libarchive to support .rar, .7z, ...
https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...
If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?
https://git.tukaani.org/?p=xz.git;a=commit;h=f9cf4c05edd14de...
- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.
- A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.
- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.
Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.
It is possible there are other flaws or backdoors in liblzma5, though.
I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.
So for 5.4.5 the tagged release and download on github differ.
It does change format strings, e.g.
There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.[Edit] formatting
Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
So it's just odd that the tags and release tarballs diverge.
liblzma5:amd64 5.4.1-0.2
I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?
My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.
Possible but unlikely.
> I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?
It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.
With our current knowledge, stable shouldn’t be affected by this.
> Running OpenSSH sshd from systemd
I think this is irrelevant.
From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively.
However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context.
> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
There's no xz code in openssh.
OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary.
> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps.
I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.
I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html
Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?
So now I actually bothered to look it up, and it turns out the actual reason is that the epoch changes what version is considered "greater", but it's not part of the .deb filename, so you still can't reuse version numbers used in the past. If you release 5.0, then 5.1, then you want to rollback and release 1:5.0, it's going to break things in the Debian archives. https://www.debian.org/doc/debian-policy/ch-binary.html#uniq...
Additionally, once you add an epoch you're stuck with it forever, while if you use 5.1+really5.0, you can get rid of the kludge when 5.2 is out. https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
Two reasons:
1. Once you bump the epoch, you have to use it forever. 2. The deb filename often doesn't contain the epoch (we use a colon which isn't valid on many filesystems), so an epoch-revert will give the same file name as pre-epoch, which breaks your repository.
So, the current best practice is the +really+ thing.
And I fail to see why bumping the epoch would ever be a problem. Using the epoch not a reason why its bad.
Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).
Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.
This particular backdoor is not shipped inside of a security patch, right?
That's a strong indication it's targeting sshd specifically.
"There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code."
"There are other checks I have not fully traced."
https://github.com/tukaani-project/xz/commit/af071ef7702debe...
Not sure what to make of this.
Most people, to find the affected versions, would either have to bisect or delve deep enough to find the offending commit. Either of which would reveal the attacker.
By not asking for the version, there is a good chance you just report "It's acting oddly, plz investigate".
[1] https://github.com/tukaani-project/.github/commit/44b766adc4...
(note: not referring to fedora here, a current fix is required. But just generally. As in, everyone is rolling out this fix, but... I mean, this codebase is poison in my eyes without a solid audit)
https://github.com/JiaT75
I hope authors of all these projects have been alerted.
STest - Unit testing framework for C/C++. Easy to use by simply dropping stest.c and stest.h into your project!
libarchive/libarchive - Multi-format archive and compression library
Seatest - Simple C based Unit Testing
Everything this account has done should be investigated.
Woha, is this legit or some sort of scam on Google in some way?:
https://github.com/google/oss-fuzz/pull/11587
edit: I have to be missing something, or I'm confused. The above author seems to be primary contact for xz? Have they just taken over?? Or did the bad commit come from another source, and a legit person applied it?
A bit confused here.
Maybe their account is compromised, maybe the username borrows the identity of an innocent person with the same name.
Focus on the code, not people. No point forming a mob.
(e: post above was edited and is no longer directed at the person. thanks for the edit.)
An account that has introduced a backdoor is not the same thing as an account who committed a bug.
Sometimes the distinction is not meaningful, but better safe than sorry.
I would now presume this person to be a hostile actor and their contributions anywhere and everywhere must be audited. I would not wait for them to cry 'but my bother did it', because an actual malicious actor would say the same thing. The 'mob' should be pouring over everything they've touched.
Audit now and audit aggressively.
$ host xz.tukaani.org
host xz.tukaani.org is an alias for tukaani-project.github.io.
And originally it was not:
$ host tukaani.org
tukaani.org has address 5.44.245.25 (seemingly in Finland)
It was moved there in Jan of this year, as per the commit listed in my prior post. By this same person/account. This means that instead of Lasse Collin's more restrictive webpage, an account directly under the control of the untrusted account, is now able to edit the webpage without anyone else's involvement.
For example, to make subtle changes in where to report security issues to, and so on.
So far I don't see anything nefarious, but at the same time, isn't this the domain/page hosting bad tarballs too?
Hetzner?
0: https://tukaani.org/about.html
Seems like an attempt to get 90 days of "use" of this vulnerability after discovery. If they only had checked performance before!
For example, their email is only registered to GitHub and Twitter. They haven't even logged into their Google account for almost a year. There's also no history of it being in any data breaches (because they never use it).
Burn the witch.
https://github.com/search?q=repo%3Alibarchive%2Flibarchive+j...
It does look innocent enough though. Let's hope there's no unicode trickery involved...
https://github.com/libarchive/libarchive/commit/e37efc16c866...
tar.exe was added to Windows this January, sourced from libarchive: https://learn.microsoft.com/en-us/virtualization/community/t...
Unlike the GNU tar I'm used to, it's actually a "full fat" command line archiving tool, compressing & decompressing zip, xz, bz2 on the command-line - really handy :-O
The PR is pretty devious.
JiaT75 claims is "Added the error text when printing out warning and errors in bsdtar when untaring. Previously, there were cryptic error messages" and cites this as fixing a previous issue.
https://github.com/libarchive/libarchive/pull/1609
However it doesn't actually do that!
The PR literally removes a new line between 2 arguments on the first `safe_fprintf()` call, and converts the `safe_fprintf()` to unsafe direct calls to `fprintf()`. In all cases, the arguments to these functions are exactly the same! So it doesn't actually make the error messages any different, it doesn't actually solve the issue it references. And the maintainer accepted it with no comments!
It does remove the safe prefixes... But it also adds one print statement to "strerror()", which could plausibly give better explanations for the error code...
The only suspicious thing here is the lack for safe_ prefix (and the potential for the strerror() function to already be backdoored elsewhere in another commit)
When the trap is in place deploy a crafted package file that appears invalid on the surface level triggers this trap. In that moment fetch the payload from the (already opened) archive file descriptor, execute it, but also patch the internal state of libarchive so that it will process the rest of the archive file as if nothing happened, and the desired outcome also appearing in the system.
https://github.com/bytecodealliance/wasmtime/commits?author=...
They've submitted little documentation tweaks to other projects, too; for example:
https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...
I don't know whether this is a formerly-legitimate open source contributor who went rogue, or a deep-cover persona spreading innocuous-looking documentation changes around to other projects as a smokescreen.
He could be doing the same thing for other reasons; nobody really digs into anything very deep so I could see someone handing over co-maintenance to a project based on a decent looking Github graph and some reasonability.
They made themselves the primary contact for xz for Google oss-fuzz about one year ago: https://github.com/google/oss-fuzz/commit/6403e93344476972e9...
- Jia Tan <jiat75@gmail.com>
- jiat75 <jiat0218@gmail.com>
``` amap = generate_author_map("xz")
```I don't see anything at https://sourcegraph.com/search?q=context:global+author:jiat0...
(I still think it's like... 60% a typo? don't know)
Anyhow, other people called the CCing of JiaT75 by Lasse suspicious:
https://news.ycombinator.com/item?id=39867593
https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...
Someone pointed out the "mental health issues" and "some other things"
https://news.ycombinator.com/item?id=39868881
https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...
Lasse is of course a Nordic name, and the whole project has a finnish name and hosting
https://news.ycombinator.com/item?id=39866902
If I wanted to go rogue and insert a backdoor in a project of mine, I'd probably create a new sockpuppet account and hand over management of the project to them. The above is worringly compatible with this hypothesis.
OTOH, JiaT75 did not reuse the existing hosting provider, but rather switched the site to github.io and uploaded there old tarballs:
https://github.com/tukaani-project/tukaani-project.github.io...
If JiaT75 is an old-timer in the project, wouldn't they have kept using the same hosting infra?
There are also some other grim possibilities: someone forced Lasse to hand over the project (violence or blackmailing? as farfetched as that sounds)... or maybe stole Lasse devices (and identity?) and now Lasse is incapacitated?
Or maybe it's just some other fellow scandinavian who pretended to be chinese and got Lasse's trust. In which case I wish Lasse all the best, and hope they'll be able to clear their name.
Is the same person sockpuppeting Hans Jansen? It's amusing (but unsurprising) that they are using both german-sounding and chinese-sounding identities.
That said, I don't think it's unreasonable to think that Lasse genuinely trusted JiaT75, genuinely believed that the ifunc stuff was reasonable (it probably isn't: https://news.ycombinator.com/item?id=39869538 ) and handed over the project to them.
And at the end of the day, the only thing linking JiaT75 to a nordic identity is a nordic racist joke which could well be a typo. People already checked the timezone of the commits, but I wonder if anyone has already checked the time-of-day of those commits... does it actually match the working hours that a person genuinely living (and sleeping) in China would follow? (of course, that's also easy to manipulate, but maybe they could've slip up)
Anyhow, I guess that security folks at Microsoft and Google (because of JiaT75 email account) are probably going to cooperate with authorities on trying to pin down the identity of JiaT75 (which might not be very useful, depending on where they live).
No, it doesn't:
https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvSG91c...
The vast majority of their Github interactions are between 12.00 UTC and 18.00 UTC
https://news.ycombinator.com/item?id=39874621
> He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.
That project just includes some metadata about a bunch of sample projects, and it links directly to a mirror of the xz project itself:
https://github.com/se-sic/VaRA-Tool-Suite/blob/982bf9b9cbf64...
I assume it downloads the project, examines the git history, and the test then ensures that the correct author name and email addresses are recognized.
(that said, I haven't checked the rest of the project, so I don't know if the code from xz is then subsequently built, and or if this other project could use that in an unsafe manner)
The google account: "Couldn't find your Google Account"
The email: "50 5.1.1 The email account that you tried to reach does not exist"
But then when you try to register it says it's taken.
Was it disabled?
I work on OSS-Fuzz.
As far as I can tell, the author's PRs do not compromise OSS-Fuzz in any way.
OSS-Fuzz doesn't trust user code for this very reason.
Fuzzing isn't really the best tool for catching bugs the maintainer intentionally inserted though.
After all, if it hadn't had a performance regression (someone could submit a PR fixing whatever slowed it down, heh) it still wouldn't be known.
> Versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key . The older releases have been signed with Lasse Collin's OpenPGP key .
It must be assume that before acquiring that privilege, they also contributed code to project. Probably most was to establish respectable record. Still could be malicious code going back someways.
Looks more likely a fake identity than compromised account.
Regardless of how likely you think it is, finding a social media footprint would be useful information. Seek information first, reach conclusions second.
https://char.tw/blog/post/24397301
https://forum.babyhome.com.tw/topic/167439
https://bmwcct.com.tw/forums/thread1828.html
- All the posts are from 2004/2006. - "jiat" can be abbreviation for many common Chinese names.
That, and I used to contribute to various games (forks of ioquake3) when I was a teen and I wanted to keep my real name private.
That's just what someone with a false identity would say.. get him boys!
The biggest /S
But wait, 2021 is his active year, but he missed almost all Aug. Is he on holiday? Who can have such a long holiday? What i can think is a solider who has a long vacation (探亲假). So let's guess he is a solider then it's sense that he worked on Spring Holiday because they need on duty. Let's double check again, if he is a solider, then they will have a holiday on every Aug. 1 because it's liberation army day. I check and no commits on all 4 years Aug. 1.
https://github.com/JiaT75
https://twitter.com/JiaTan1337/status/1774931375994319244
kind of interesting also to see this account was set up ~2 months ago. if it's a troll, it's a somewhat poor joke.
It looks like gettext may be containing a part of their attack infrastructure.
https://github.com/microsoft/vcpkg/pull/37199#pullrequestrev...
https://github.com/microsoft/vcpkg/pull/37356/files#diff-e16...
https://github.com/MonicaLiu0311
This is from Gnulib, which is used by Gettext and other GNU projects. Using 'setlocale (0, NULL)' is not thread-safe on all platforms. Gnulib has modules to work around this, but not all projects want the extra locking. Hence the name '-unsafe'. :)
See: https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg001...
Timeline matches and there is a sudden switch of maintainer. And they add dependency to xz!
[0]: https://git.alpinelinux.org/aports/stats/?period=y&ofs=10
Or it could in theory be malware authors (ransomware, etc). However these guys tend to aim at the low hanging fruits. They want to make a buck quickly. I don't think they have the patience and persistence to infiltrate an open source project for 2 long years to finally gain enough trust and access to backdoor it. On the other hand, a state actor is in for the long term, so they would spend that much time (and more) to accomplish that.
So that's my guess: Jia Tan is an employee of some intelligence agency. He chose to present an asian persona, but that's not necessarily who he truly represents. Could be anyone, really: Russia, China, Israel, or even the US, etc.
Edit: given that Lasse Collin was the only maintainer of xz utils in 2022 before Jia Tan, I wouldn't be surprised if the state actor interfered with Lasse somehow. They could have done anything to distract him from the project: introduce a mistress in his life, give him a high-paying job, make his spouse sick so he has to care for her, etc. With Lasse not having as many hours to spend on the project, he would have been more likely to give access to a developer who shows up around the same time and who is highly motivated to contribute code. I would be interested to talk to Lasse to understand his circumstances around 2022.
I agree that this is likely a state actor, or at least a very large & wealthy private actor who can play the long game…
Most likely this is not the first backdoor, just the first one to be discovered, so it wasn't two years of work until there were results.
But I still agree that he's probably a state actor.
In fact, that'd be the best form of deep cover. It'll be interested to watch as people more knowledgable than I pour over every single commit and change.
There could be other backdoors for other targets.
(1) A backdoor like this one, which isn't really about its core functions, but about the fact that it's a library linked into critical code, so that you can use it to backdoor _other things_. Those are complex and tricky because you have to manipulate the linking/GOT specifically for a target.
(2) Insert an exploitable flaw such as a buffer overflow so that you can craft malicious .xz files that result in a target executing code if they process your file. This is a slightly more generic attack vector but that requires a click/download/action.
Not every machine or person you want to compromise has an exposed service like ssh, and not every target will download/decompress a file you send to them. These are decently orthogonal attack vectors even though they both involve a library.
(Note that there's as yet no evidence for #2 - I'm just noting how I'd try to leverage this to maximum effect if I wanted to.)
these files are also useful to check that the library we just built works correctly. but they aren't necessary for installation.
we may have more sophisticated procedures that will allow us to use some parts of distribution only for tests. This may significantly reduce an attack vector - many projects have huge, sophisticated testing infrastructure where you can hide the entire Wikipedia.
Yeah, the actor part seems unnecessary now.
People could also just get tired after years of active maintainership or become busier with life. Being the sole maintainer of an active open source project on top of work and perhaps family takes either a lot of enthusiasm or a lot of commitment. It's not really a given that people want to (or can) keep doing that forever at the same pace.
Someone then spots the opportunity.
I have no idea what the story is here but it might be something rather mundane.
https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...
> With your current rate, I very doubt to see 5.4.0 release this year. The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?
https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.h...
The last two sentences really make it look as if he were trying to pressure the original author.
"Your efforts are good but based on the slow release schedule it will unfortunatly be years until the community actually gets this quality of life feature."
"Patches spend years on this mailing list. 5.2.0 release was 7 years ago. There is no reason to think anything is coming soon."
"With your current rate, I very doubt to see 5.4.0 release this year. The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"
"Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this."
"Is there any progress on this? Jia I see you have recent commits. Why can't you commit this yourself?"
"Over 1 month and no closer to being merged. Not a suprise."
The stuxnet malware, which compromised Siemens industrial controls to attack specific centrifuges in uranium enrichment plants in Iran, is a counterexample to that.
But, anyway, I'm sure we can find other counter-examples.
If a government went to this much effort to plant this vulnerability, they absolutely have targets in mind - just like they did when they went to the effort of researching (or acquiring) four separate Windows zero-days, combining them, and delivering them...
Couldn't the account that committed the backdoor have been compromised recently?
I've never heard anyone claim that Dual_EC_DRBG is most likely not intentionally backdoored, but there's literally no way to confirm because of how its written. If we can't analyze intention from the code, we can look at the broader context for clues. The NSA spent an unusual amount of effort trying to push forward an algorithm that kept getting shot down because it was slower than similar algorithms with no additional benefits (the $10 million deal specified it as a requirement [1]). If you give the NSA the benefit of the doubt, they spent a lot of time and money to... intentionally slow down random number generation?!
As an American, I'd prefer a competent NSA than an incompetent NSA that spends my tax dollars to make technology worse for literally no benefit...
[1] https://www.reuters.com/article/us-usa-security-rsa-idUSBRE9...
To be clear, the method of attack was something that had been described in a paper years earlier, the NSA literally had a program (BULLRUN) around compromising and attacking encryption, and there were security researchers at NIST and other places that raised concerns even before it was implemented as a standard. Oh, and the NSA paid the RSA $10 million to implement it.
Heck, even the chairman of the RSA implies they got used by the NSA:
In an impassioned speech, Coveillo said RSA, like many in industry, has worked with the NSA on projects. But in the case of the NSA-developed algorithm which he didn’t directly name, Coviello told conference attendees that RSA feels NSA exploited its position of trust. In its job, NSA plays two roles, he pointed out. In the information assurance directorate (IAD) arm of NSA, it decides on security technologies that might find use in the government, especially the military. The other side of the NSA is tasked with vacuuming up data for cyber-espionage purposes and now is prepared to take an offensive role in cyber-attacks and cyberwar.
“We can’t be sure which part of the NSA we’re working with,” said Coviello with a tone of anguish. He implied that if the NSA induced RSA to include a secret backdoor in any RSA product, it happened without RSA’s consent or awareness.
https://www.networkworld.com/article/687628/security-rsa-chi...
There's an implicit "always" in their second sentence, if you're confused by the wording. They aren't positing the equivalent of the guard that only lies.
> It is naive to think that US intelligence agencies will act in the best interest of US citizens or companies.
With an example of them doing exactly that.
Is there something more productive that I could have replied with? (I know I could have been less snippy, but I think being snippy is fair there.)
You coulda just pointed out that just because they did right in the case of DSA, doesn't mean we should actually ever trust them, which I would agree is the correct stance.
Mostly I think that story is neat and wanted people to know about it, so I asked a question as a performative writing technique.
"If you're confused by the wording" was definitely condescending, but I think interpreting guinea-unicorn's post that way doesn't make sense. Even in your reply you didn't say you think it's the right interpretation, just that someone might believe the NSA could "only be up to evil". That followup gives the impression you were giving an FYI for readers. Which is nice to do, but then the "what about" doesn't fit.
So all of that is to say the words "what about" felt like you were deciding to read their post in an unfair way.
I'm happy to listen to an alternate explanation! But you ignored my request for why you said that, and I'm honestly kind of confused as to why that's what set you off.
So overall I think I think my first post can come across as fighty but I don't think the followups should suggest I'm making things fighty. I think my response to 2OEH8eoCRo0 was fine given the way they were ignoring half of the four sentences I had typed.
I can see some arguments that might persuade the NSA to run an attack like this
You say that as if members of US government agencies didn't plot terror attacks on Americans (Operation Northwood), steal the medical records of American whistleblowers (Ellsberg), had to be prevented from assassinating American journalists (Gordon Liddy, on Jack Anderson), collude to assassinate American political activists (Fred Hampton), spy on presidential candidates (Watergate), sell weapons to countries who'd allegedly supported groups who'd launched suicide bombing attacks on American soldiers (Iran-Contra), allow drug smugglers to flood the USA with cocaine so that they could supply illegal guns to terrorists abroad on their return trip (Iran-Contra again) and get caught conducting illegal mass-surveillance on American people as a whole (Snowden). Among others.
It's super-naive to suggest that government agencies wouldn't act against the interest of American citizens and companies because there might be consequences if they were caught. Most of the instances above actually were instances where the perpetrators did get caught, which is why we know about them.
You kind of lost the thread when you say, “act against the interests of American citizens and companies”. Bro, literally anyone could be using xz, and anyone could be using Red Hat. You’re only “acting against Americans” if you use it against Americans. I don’t know who was behind this, but a perfectly plausible scenario would be the NSA putting the backdoor in with an ostensibly Chinese login and then activating on machines hosted and controlled by people outside of the US.
Focusing on a specific distro is myopic. Red Hat is popular.
There's a term for that: NOBUS (https://en.wikipedia.org/wiki/NOBUS). It won't surprise me at all if this backdoor can only be exploited if the attacker has the private key corresponding to a public key contained in the injected code. It also won't surprise me if this private key ends up being stolen by someone else, and used against its original owner.
And that is exactly why backdoored encryption is bad.
The HN crowd has come a long way from practically hero-worshipping Snowden to automatically assuming that 'state actor' must mean the countries marked evil by the US.
By comparison America is actually quite timid compared to other countries e.g. UK and the widespread CCTV network.
Now, GCHQ is no better than the NSA for that either, but I don't think CCTV is a good comparison.
And yes, most of modern supporters of Wikileaks / Assange / Snowden / etc, chanting 'release Assange' and 'pardon Snowden' are useful idiots in hands of tyrannies like BRICS club.
I think the best reason to doubt USG involvement is the ease with which somebody discovered this issue, which is only a month or two old. I feel like NSA etc. knows not to get caught doing this so easily.
Edit: Also, Github has suspended both accounts. Perhaps they know something we don't.
[1] https://news.ycombinator.com/item?id=39865810#39866275
[1] https://news.ycombinator.com/item?id=39867593
[2] https://news.ycombinator.com/item?id=39866936
Are you looking at the same repositories I am? He's made 88 commits to xz in that time period, two-thirds of the total.
Lol, what
> wants to cast a wide net to attack as many as possible. So that fits the profile of a government intelligence agency
That's quite backwards. Governments are far more likely to deploy a complex attack against a single target (see also: Stuxnet); other attackers (motivated primarily by money) are far more likely to cast a wide net.
Governments are well known to keep vulnerabilities hidden (see EternalBlue). Intentionally introducing a vulnerability doesn’t seem that backwards tbh
BYW,I had a classmate who used to play DOTA1(on war3) under this name at the University of Science and Technology of China a long time ago, and this was his first girlfriend name (maybe) . His father was a high-ranking official. Then he joined the parent department of the Internal Security Detachment, a secret service that has gained a lot of power in the last few years. I hope I'm not awake . lol.
Discussing commits that the other author has since reverted, IFUNC change with Project Zero tests, a focus on embedded, etc.:
https://www.mail-archive.com/xz-devel@tukaani.org/msg00642.h...
Trimming security reporting details:
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=af071ef7702...
https://github.com/snappyJack/CVE-request-XZ-5.2.5-has-denia...
He understood the software architecture quite early on while working on the following repository. He connected the dots from his other projects and went rogue. (probably to benefit from crypto?). Take a look at his other repositories and code style and recent likes on github. Is he our Jia Tan?
Secondly, the use of English is not consistent in what should be from typical Indian. He should be from a foreign background or a very reputed English medium.
The language though seemingly simple for a native English speaker but it seems in this case; a person whose first language: likely is not English.
It is possible that Grammarly or auto correct could have been used to write these. But can't be certain of anything stated above.
I do think that this is a sabotage account with 60% chances unless Mr. Kumar comes out clean, publicly. He is likely a state sponsored actor.