Expert level opsec - very rarely make deliberate "mistakes" by having an inconsistent timezone, or tunneling through someone's compromised home device instead of a VPN, to throw adversaries on wild goose chases.
Most (but not all ) VPN providers keep logs and payment info that are subpoenable. You could use something like Mulvad with Lightning Network payments, but I am not sure that even that is fully anonymous.
The Witopia VPN that he used for IRC [1] is US based: https://www.personalvpn.com/contact-us/ and they don't mention neither LN payments nor not keeping logs.
Departure postal code could be an area with like 1 million people. And if it is a nation state I am sure they could send the mail from another country.
I mean, even if I live in Manhattan, I can easily take a subway train to an area with a different zip code and mail it there. And that's almost the easiest thing you could do -- if you want to hide where the mail is from, it is trivial.
Dense urban areas are often blanketed by a range of spectrum sensors for the purpose of retroactive correlation (e.g Palantir) with other metadata sources.
There's... actually very little that stops you from sending mail from a non-local postal code.
I've occasionally sent packages postmarked as being from one zipcode from another; as long as it's in the same region, much of the postal processing doesn't care so much.
How do you do that? Walk up to the postal counter, ask them to postmark it, then ask for it back, drive to another post office and slip it in their outgoing pile?
Semi-Presorted post can be acquired pretty easily. Places like Shippo and PirateShip offer it. As long as you put enough money into the postage paid, the post office doesn't really give a shit if it comes out of somewhere weird. There is no requirement that "return address" and "sent from" area are the same so long as they're within the same postal zone.
This is why AMZN packages have a return address in Vegas or similar sometimes.
I had a hard time choosing which comment to select to reply, so I chose yours since it's higher up. Apologies if it's irrelevant.
I don't know why most people assume that hackers even bother with stolen credit cards in the first place. I mean, they sure do, but those are your average Joes in the business of refund reshipping and other types of scams.
Those who want the maximum anonymity don't even bother with buying anything. It's as simple as going to one of the popular websites who leak databases, setting up OpenBullet software or spending anywhere from 1 to 5 hours writing custom mail:pass validators to spam requests to either API or login form through (once again) leaked proxies, etc. using leaked credentials. Or simply going into one of those threads titled 'x100 Mullvad accounts" which have already validated accounts with anywhere from 1m pre-paid to multiple years. And there's even a bonus of not being shown as a user of this account if you do not use official App and simply load configuration manually through ovpn, etc.
And then there's proxy-chaining if you're doing something truly nefarious. It's super easy to chain multiple VPNs with few socks proxies.
People behind XZ backdoor to me look much more smarter than myself, so I would bet they took care of this angle and will be untraceable.
Shame on me for not reading more before my now removed comment. Shout-out for doing this level of analysis and guessing, as with everything in this incident, fascinating and tantalizing to wonder about.
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
> A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
> Obviously it would help avoiding detection since the backdoor has the side effect of slowing down things.
My understanding was that this was the result of a bug? But I understand that part less than any of it, so I well could be wrong.
>And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
Now that, strikes me as more likely. Maybe they wanted to be relatively sure that they wouldn't be detected and needed to select a test matrix to test against (SELinux/landlock config across rpm/deb distros, versus across every distro.)
I guess what I'm getting at is... did they have an intended set of targets and knew they were running deb/rpm, on top of any other factors?
I don't think it was specific effort to exclude others, they took efforts to ensure it happened at packaging time, to reduce chances of detection. So eg other xz devs didn't notice it when compiling it, and so on.
> A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
The payload only works on systems that connect their sshd to systemd-notify -- which is not true for many Linux distributions beyond Debian-based and RH-based anyway (e.g. Arch Linux doesn't do this).
Poor Jia. Two whole years of work down the drain. If you're reading this Jia, remember that you miss 100% of the shots you don't take. Chin up, brother.
Yes, unironically, that, being a huge leak, really convinces me that we all have multiple RCEs on each of our systems. I mean, we already kinda knew that, of course, but even now that I'm typing this it's a bit hard for me to actually believe it. But realistically I think it's pretty much a proven fact. This level of sophistication… what did I even expect? Of course well-payed teams of high-level professionals are working in this area for a very long time by now. This is basically like professional football players competing against office workers in their late thirties who like to kick a ball on weekends. I am comforting myself by thinking that something a bit more high-level, written in python or even golang, or, oh hell, maybe even rust would be more a bit more transparent for other contributors… but even this is probably a lie. And if it wouldn't be, surely tons of super-popular low-level packages written in C/C++ are as good as incomprehensible for somebody who isn't specifically security-auditing this.
I never understood and continue to be baffled by the naïveté of FOSS people giving commit access to core projects to unknown parties. Assume they're a state actor or a crazy person, until proven otherwise.
That effectively means you cannot share commit rights with anybody. How am I supposed to vet someone as not a state actor? Any plausible test could be thwarted by a motivated individual.
A state actor is potentially the only person financially supported enough and willing to do this.
"You're in Zurich this weekend? Oh wow, me too. I'm holidaying with my wife. Let's grab lunch" as they proceed to prepare their military expense form for two business class tickets for them and their coworker
Setting aside the gravity of the situation, I'm 100% gripped by the "mystery novel" side of this. Seeing the internet piecing together the Who? the How? an the Why? is fascinating.
I dont know what the XZ backdoor is about but, to assume someone putting backdoors into software would work usual working days and hours has to be very naïve. The whole premise of the article is based on a false assumption IMO.
If you’ve missed probably the largest cyber security story since stuxnet, and arguably bigger than that, I suggest you start looking at the last few days.
That’s a stretch. Stuxnet was the first acknowledged state cyber attack, utilized multiple zero days, and destroyed nuclear weapons manufacturing facilities. Bigger in scope sure, but bigger unconditionally? I don’t know about that.
And that would likely be precious few, at least today. Only a teeny tiny percentage of servers out there were updated to a vulnerable version. And an even smaller percentage likely had their sshd port exposed to the public internet.
Sure, if the backdoor hadn't been found for a couple more years, the impact would have been much higher, as the backdoored version would have made it into actual current releases of the popular server distros, and companies gradually upgraded.
From my point of view, from what we know today, it is bigger than Stuxnet because:
Stuxnet: aimed to delay one nuclear facility that was still being built
SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world. With SSH's track record, a lot of people choose to trust it as their internet-facing access protocol
I expect that whoever made this would have picked their targets carefully to avoid revealing the backdoor, so probably they'd aim (at least at first) at a handful of businesses of interest (advanced chip manufacturing, say) and governments, rather than causing tangible widespread issues in some way. Theoretically, though, (perhaps upon being discovered) if they'd just drop `rm -rf && poweroff` on all reachable systems (perhaps having it spread into networks in a worm-like fashion, setting a timer for 30 seconds so that it can propagate), most computer-based systems would just stop working (either killed themselves, or from failing routers or other systems they relied on), and lots of them would lose data because their backups either weren't working or were also impacted. Consider just how much involves a computer today, from infrastructure to hospitals. That's a whole lot more impact than a delayed nuclear program due to some unpublished Windows exploits
What modulates this potential impact is the question of how many important systems run something other than the OSes they were currently in the process of targeting (Debian, Ubuntu, and Fedora afaik), how many servers require a VPN (or similar) to access ssh and have no outward-facing ssh server anywhere on their network, and how many years it would have taken to uncover (more and more systems would have been running this over time)
It's not clear that the distribution mechanism and many of the other requirements would be so broadly met that "access to most servers on the internet" is a correct description of the scope.
Or that, once in the wild, that the author would have their pick of the litter when choosing targets. It's also true that the exploit has several kill switches in it, and so even vulnerable servers could be protected by simply setting an environment variable.
Which, honestly, makes this entire attack all the more strange. It had incredibly unclear value and longevity. It makes me wonder if the author didn't have different, parallel aims, or if this wasn't a type of practice run for a larger attack in the future.
Since they would have "their pick of the litter" why did they then include kill switches?
A scary scenario is a rerun of the Viasat hack https://en.wikipedia.org/wiki/Viasat_hack
I.e. wipe all servers worldwide at the morning of an attack, but before that you install the kill switches on your servers and those of your allies.
To make it harder to detect and reverse engineer. There's a plethora of disabling functionality in the exploit. This is bad, but the original post is being hyperbolic.
> SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world.
No, absolutely not; that didn't actually happen. Most servers on the planet that are running x86_64 Linux are probably not running a rolling-release distro based on .deb or .rpm. Rolling-release (or a beta version of the next OS release) is required because I can't imagine one of them updating to a new version of a library like this until the next major release of their OS, and .deb/.rpm because those are the only build environments where the backdoor would get built into the library. (And on top of that, only systems where sshd is patched to link to libsystemd.)
Also: while I will admit that there are many businesses with abysmal security practices, most will not have ssh exposed to the public internet; that'll require access to the corporate VPN as well. You note this, but fail to recognize the impact.
Even hobbyists probably aren't hit by this in large numbers. I run a VPS for a variety of things, and it runs Debian stable. Assuming this backdoor attempt was never found, it wouldn't get installed on it until mid-/late-2025, when Debian trixie is likely to be released (plus some time for me to get around to upgrading it).
I did have the affected package on my laptop, which runs Debian testing. But I don't have sshd enabled on it all the time, and even if I did, my laptop is rarely on a network without NAT (and I usually opt to tether to my phone when in public rather than use public WiFi). All the other machines on my home network are also running Debian stable, and were unaffected.
I suspect that precious few systems even had the backdoor installed on them, and of those, even fewer were accessible directly on the public internet.
Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
See modulating factors at the bottom. In this case, it didn't because it was caught before making it into a stable release, but only due to sheer luck. The story is still so much bigger than Stuxnet to me because its aim was an actually tangible impact on millions of people's lives
> Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
Agreed on that! (To be clear, I still think it's already bigger, not because of the impact it concretely had but because I perceive it to having been very close; but I can understand this/your point of view very well)
I suggest not putting a UTC+8 timezone graph as header image, when the conclusion is he's likely from Eastern Europe (not saying you should put one showing EET timezone instead).
I understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't (hence why clickbait works). I also think it's a little tasteless, if not misleading.
The whole enterprise is a little tasteless with a chance of worse. Not some substacker or tweeter speculating in itself - people are obviously free to let their speculation flag fly. But you start amplifying this stuff on an open internet messageboard and next thing you know, it's reddit 'found' the Boston bomber. The local downsides outweigh the gawking value and I say this as a dyed-in-the-wool internet gawker.
> understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't
People who don’t read the article and find themselves lacking an informed or nuanced understanding of the contents of said article are the only ones to blame here. Full stop.
That HN doesn’t have a culture of shaming those who don’t read the article is partly the fault of the rules here which finds “read the article please” to be a distraction/nuisance but don’t mind that this often results in misinformed discussion propagating toward the top of a thread.
I don't think he was from Eastern Europe, but if you want to look at UTC+0200/+0300, in Europe this only includes Finland, Baltics, Ukraine, Romania, Moldavia, and Greece. But notably if you look a bit down it also includes a good chunk of the Middle East, including Israel.
I don't know about holidays but Israel uses the same daylight savings as the mentioned Eastern Europe countries, ie switching from UTC+0200 to +0300 at around the same time - only difference is that they start daylight savings on Friday instead of Sunday the same week.
I don't know all the holidays but they have commits on Yom Kippur dates in 2021 and 2022. For 2023 it's harder to say because their commit activity is sparser.
Israelis are usually off from the evening of Friday (sundown) till the evening of Saturday (sundown) for Shabbat. Where did you get the idea that they’re off on Fridays during daytime?
In this context, middle eastern countries with Islam as the majority religion have the day off on Fridays.
Not sure why you’re downvoted. When you think of state actors, Israel comes very high (stuxnet etc), as well as the usual US/Russia/China/NK groups.
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
If the only think I knew was UTC+2 (and I don't even think I know that, I'm really cautious with assumptions about what mistakes that kind of people plausibly make), Israel would be very high on the list. But FWIW, 25Dec is a Catholic holiday, and even though it's mentioned on Wikipedia page about Israel holidays, I'm not sure how official that is. I mean, it doesn't seems to be, like, a real Israeli holiday.
I think no commits on December 25 is not enough to go by. He's been active for about 2 years, so that's what, 1-2 Christmases? I assume he doesn't commit literally every day. So it could be a coincidence that he didn't on those particular 1-2 days. Also, in many Orthodox Christian majority countries including those in UTC+2 they don't celebrate on 12/25.
But doesn't Israel also not follow a typical work week? Eg. No commits on Friday afternoon?
Yes, that's what I mean as well: I don't really think this analysis is any near to be considered any conclusive. It's interesting. But this is it. From what I'm seeing, there really isn't very much data. The article makes it feel like there is this huge archive of commits, that draws a picture of a guy working almost every day on these projects with some very much visible gaps in time, but there really isn't. The data is quite scarce. I would need to actually do the analysis myself to decide what I'm willing to believe, and I don't think I want to do this right now. But for now, I don't think we really have anything.
…What I would really want is some input from Microsoft. I don't know what they can reveal, but I believe they must have quite a bit more data, than us. Even speaking about times, well, he must have been using Github more than to just commit stuff. And I'd imagine they have some logs. Also, it's very fair to assume than he was using VPN all the time, but it's also fair to assume that he wouldn't accidentally slip like that, revealing his real time zone in 9 commits. So, yeah, for sure there are people who have waay more data than me, and probably know how to use it properly better than me too. Not sure they will be willing to share, though.
No need to bother with a tracking image. Just scour the message headers of any message sent in the past--there's decent chance there's a few more dates lurking in those headers, and even an IP address or two.
Years ago they would include the IP of the end user. Gmail and hotmail stopped including that more than a decade ago. But they both used to. I think Gmail stopped this practice years ahead of Hotmail.
I have a relative with mental health problems who lives on the street, and he sometimes writes me from public libraries -- kind of luckily for family members who are concerned about him, the webmail service he uses still includes that info in headers. So I usually know which public library he writes from.
It's not just that there's only two Christmases. It's that Christmas fell on Sunday in 2022 and Monday in 2023, and the article already established the attacker mainly worked Tuesday-Friday. The note about the attacker never working on New Year has the same problem (New Year's Eve/Day always lines up with Christmas Eve/Day, so would not have hit the Thu-Fri window either).
December 25 and and Jan 1 are regular work days in Israel, with the legal option that someone can't be refused to take those days off, but have it deducted from their accrued vacation time (there are other days that one can't be refused to take off, but if no available vacation days, have to be offered leave without pay for those days).
NK are one if the most prolific state actor countries. They aren't UTC+2 though.
Greece and Finland aren’t prolific. I’m sure they have the resources, many lone people would have. I don’t think they have the motives though. The US, Russia, China, NK and Israel have the motive, opportunity and track record. I wouldn’t rule out Ukraine either (and Ukraine is GMT+2)
I still think that an individual is the most likely candidate though.
“Trust no one! The minute God crapped out the third caveman, a conspiracy was hatched against one of them! ” - Gen. Hunter Gathers, OSI (The Venture Bros.)
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
> Never, ever install bleeding-edge software in production, for any reason.
But for any X which is mainstream today someone was always the first of X. And even then being the second or third is still cutting edge. For a certain kind of company it makes sense to play this way, but if it were everyone then we'd have a tragedy.
That's why you run X in your sandbox/QA/testing/whatever-you-call-it environment, safe from the prying eyes of the public internet and the private data of users. Once it's all good there, and the community has come to a consensus about it being safe and ready-for-release, that's when you can put it wherever you want.
99.9% of releases for packages like this are boring bugfixes and stuff, not earth-shattering new features. You're not at a competitive disadvantage for not having taken this version of xz and having routed all of your traffic through it.
For years I've had `gc` in my terminal mapped to `TZ=UTC0 git commit` for exactly this reason. No need to change system times or git settings. (though perhaps a better way in global config?)
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
Time zone is purely a displayed time consideration. I'm not bothered that the display time on my computer is different from the time zone I'm actually in.
Same. My phone automatically changes, and I'll manually change my watch, but when I travel, I leave my laptop in my home time zone. Part of it is I just can't be bothered, and part of it is I want an easy way to know what time it is at home without having to fiddle with a world clock.
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
The "hacker" stereotype is certainly stale, but subbing "young person" (and implying not "old person") is equally silly.
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
> There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
I knew most of the stuff in that article. If I wake up at 5am on full alert, mind racing, I read for a couple of hours and then go back to sleep when it's possible. I know that I can't function well on less than 7 hours, so I don't try.
The article seems to be taking about people on the verge of entering "the facility", on their slow slide into dementia, who are making do despite not getting enough sleep. I'll get back to you when I hit 70.
No, stereotypes aside, hackers who code diligently tend to wrap their "actual work" schedule around their external obligations, and while the morning is generally filled with distractions like dealing with "morning people", after lunch (an on into the evening) there are generally far fewer interruptions. I can honestly get much more done with an uninterrupted four hours than I can in the entire eight office hours of random phone calls and emails coming in, and I am no longer a "young person", and I'll do some stuff at night just because I know it'll take half the time if I'm not interrupted and I've had a few hours to at least intermittently mull it over. If you're having peaceful mornings, I envy you.
If we're putting stereotypes aside, wouldn't we also put aside the word "hackers"?
In the xz case, the term may be accurate, but we don't know the identity of the culprit(s) and have no empirical basis for even speculating about the work schedules of such people.
> If you're having peaceful mornings, I envy you.
Well, I don't work in an office. I work alone. Like a stereotypical "hacker".
> Generally, anonymity in the free software sphere is a good thing: software is inherently based on accomplishment and merit, and there is no reason to know anything about a person’s identity.
This is an interesting take. Most software engineers I know love to show off their FOSS contributions. It's a place to do what _you_ want to do, free of budgets and MBA interference. I guess in certain parts of the world you probably want to keep that sort of skill set under wraps, though.
I think personality matters a lot here. One person may flaunt their open source contributions to their colleagues because they like the kudos. Another person may quietly commit and never publicize themselves because they don’t like being the center of attention. Both can even be true of the same person at different points in their careers!
It would be interesting to see how the commit timestamps change around time change. If he has a job in a country that observes DST they would likely show an abrupt change.
Saying "Eastern Europe" as opposed to "Moscow" is a little coy, isn't it? They're in the same time zone and one seems likelier than the other, to put it one way.
Edit: speaking of, these guys might also have some time pressure at the moment with regards to hacking stuff lol. Kind of fits the bill with what we know about the hack
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
> Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
I would remind people that it is not difficult at all to have someone's sleep schedule match another country, particularly when we're dealing with what seems to be a state-sponsored attack. Times are a red herring - don't be fooled.
Everyone here is acting as if it's impossible to script git commits or email replies as a scheduled task or even put someone on the night shift to click send - to give the appearance of being somewhere you're not.
There was WAY more than one person involved here. Every email, commit, etc. was a deliberate task that was carefully planned.
Would be nice if whoever made the analysis would attach the actual data in csv or something. I mean, it's a bit handier to overview than actually cloning the repo(s) ang using git log, if I'm not gonna dive deep.
Do you understand the time format there? It lists things as -700 or -800 for me, is that because of my local time zone (Chicago) vs. the mailing list (Finland?).
I think mail-archive.com just shows time stamps in UTC-0700 or UTC-0800 regardless of where you are. That's how time stamps appear on all of the few mailing lists I checked out there, regardless of the sender of the message.
If some nation state actor dis this, I imagine they couldve easily modified times of their commits, takin into consideration holidaya and such. Also, nation state actors have tons of resources. They could have people waiting to make commits or schedule it with something like cron.
192 comments
[ 3.9 ms ] story [ 224 ms ] threadEdit: it’s a meme. Apparently. I feel like an old man.
> illia-git This user is suspected to be a part of an online ransomware organization. Please report any suspicious activity to GitHub security. Poland
https://knowyourmeme.com/memes/discord-user-is-a-suspected-t...
It’s quite plausible that they didn’t manage perfect vpn usage every single time.
The thing is you only need to make a mistake once.
Jia Tan probably used a vpn though - we know that they did for accessing IRC (source: https://boehs.org/node/everything-i-know-about-the-xz-backdo...)
The Witopia VPN that he used for IRC [1] is US based: https://www.personalvpn.com/contact-us/ and they don't mention neither LN payments nor not keeping logs.
1. "~jiatan@185.128.24.163" https://boehs.org/node/everything-i-know-about-the-xz-backdo...
Or that they store something that allows correlating an IP+time to a Mullvad account.
When a VPN provider isn't cooperative, metadata can be sought upstream.
Mullvad deserves much credit for accepting non-digital payments, which increases the cost of deanonymization.
I've occasionally sent packages postmarked as being from one zipcode from another; as long as it's in the same region, much of the postal processing doesn't care so much.
There's also remailers and forwarders.
This is why AMZN packages have a return address in Vegas or similar sometimes.
I don't know why most people assume that hackers even bother with stolen credit cards in the first place. I mean, they sure do, but those are your average Joes in the business of refund reshipping and other types of scams.
Those who want the maximum anonymity don't even bother with buying anything. It's as simple as going to one of the popular websites who leak databases, setting up OpenBullet software or spending anywhere from 1 to 5 hours writing custom mail:pass validators to spam requests to either API or login form through (once again) leaked proxies, etc. using leaked credentials. Or simply going into one of those threads titled 'x100 Mullvad accounts" which have already validated accounts with anywhere from 1m pre-paid to multiple years. And there's even a bonus of not being shown as a user of this account if you do not use official App and simply load configuration manually through ovpn, etc.
And then there's proxy-chaining if you're doing something truly nefarious. It's super easy to chain multiple VPNs with few socks proxies.
People behind XZ backdoor to me look much more smarter than myself, so I would bet they took care of this angle and will be untraceable.
[1] https://i.redd.it/obvqaa0lhn841.jpg
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
My understanding was that this was the result of a bug? But I understand that part less than any of it, so I well could be wrong.
>And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
Now that, strikes me as more likely. Maybe they wanted to be relatively sure that they wouldn't be detected and needed to select a test matrix to test against (SELinux/landlock config across rpm/deb distros, versus across every distro.)
I guess what I'm getting at is... did they have an intended set of targets and knew they were running deb/rpm, on top of any other factors?
The payload only works on systems that connect their sshd to systemd-notify -- which is not true for many Linux distributions beyond Debian-based and RH-based anyway (e.g. Arch Linux doesn't do this).
"You're in Zurich this weekend? Oh wow, me too. I'm holidaying with my wife. Let's grab lunch" as they proceed to prepare their military expense form for two business class tickets for them and their coworker
But I am sure JiaT didn't only work on the xz project?
It would be like spending hours baking bread only to eat it before it goes in the oven.
Start here.
https://news.ycombinator.com/item?id=39865810
That’s a stretch. Stuxnet was the first acknowledged state cyber attack, utilized multiple zero days, and destroyed nuclear weapons manufacturing facilities. Bigger in scope sure, but bigger unconditionally? I don’t know about that.
Sure, if the backdoor hadn't been found for a couple more years, the impact would have been much higher, as the backdoored version would have made it into actual current releases of the popular server distros, and companies gradually upgraded.
Stuxnet: aimed to delay one nuclear facility that was still being built
SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world. With SSH's track record, a lot of people choose to trust it as their internet-facing access protocol
I expect that whoever made this would have picked their targets carefully to avoid revealing the backdoor, so probably they'd aim (at least at first) at a handful of businesses of interest (advanced chip manufacturing, say) and governments, rather than causing tangible widespread issues in some way. Theoretically, though, (perhaps upon being discovered) if they'd just drop `rm -rf && poweroff` on all reachable systems (perhaps having it spread into networks in a worm-like fashion, setting a timer for 30 seconds so that it can propagate), most computer-based systems would just stop working (either killed themselves, or from failing routers or other systems they relied on), and lots of them would lose data because their backups either weren't working or were also impacted. Consider just how much involves a computer today, from infrastructure to hospitals. That's a whole lot more impact than a delayed nuclear program due to some unpublished Windows exploits
What modulates this potential impact is the question of how many important systems run something other than the OSes they were currently in the process of targeting (Debian, Ubuntu, and Fedora afaik), how many servers require a VPN (or similar) to access ssh and have no outward-facing ssh server anywhere on their network, and how many years it would have taken to uncover (more and more systems would have been running this over time)
Or that, once in the wild, that the author would have their pick of the litter when choosing targets. It's also true that the exploit has several kill switches in it, and so even vulnerable servers could be protected by simply setting an environment variable.
Which, honestly, makes this entire attack all the more strange. It had incredibly unclear value and longevity. It makes me wonder if the author didn't have different, parallel aims, or if this wasn't a type of practice run for a larger attack in the future.
No, absolutely not; that didn't actually happen. Most servers on the planet that are running x86_64 Linux are probably not running a rolling-release distro based on .deb or .rpm. Rolling-release (or a beta version of the next OS release) is required because I can't imagine one of them updating to a new version of a library like this until the next major release of their OS, and .deb/.rpm because those are the only build environments where the backdoor would get built into the library. (And on top of that, only systems where sshd is patched to link to libsystemd.)
Also: while I will admit that there are many businesses with abysmal security practices, most will not have ssh exposed to the public internet; that'll require access to the corporate VPN as well. You note this, but fail to recognize the impact.
Even hobbyists probably aren't hit by this in large numbers. I run a VPS for a variety of things, and it runs Debian stable. Assuming this backdoor attempt was never found, it wouldn't get installed on it until mid-/late-2025, when Debian trixie is likely to be released (plus some time for me to get around to upgrading it).
I did have the affected package on my laptop, which runs Debian testing. But I don't have sshd enabled on it all the time, and even if I did, my laptop is rarely on a network without NAT (and I usually opt to tether to my phone when in public rather than use public WiFi). All the other machines on my home network are also running Debian stable, and were unaffected.
I suspect that precious few systems even had the backdoor installed on them, and of those, even fewer were accessible directly on the public internet.
Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
See modulating factors at the bottom. In this case, it didn't because it was caught before making it into a stable release, but only due to sheer luck. The story is still so much bigger than Stuxnet to me because its aim was an actually tangible impact on millions of people's lives
> Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
Agreed on that! (To be clear, I still think it's already bigger, not because of the impact it concretely had but because I perceive it to having been very close; but I can understand this/your point of view very well)
I understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't (hence why clickbait works). I also think it's a little tasteless, if not misleading.
Disclaimer: I'm a Chinese.
the rest is mostly speculation about where else they could have been from given holidays and times
People who don’t read the article and find themselves lacking an informed or nuanced understanding of the contents of said article are the only ones to blame here. Full stop.
That HN doesn’t have a culture of shaming those who don’t read the article is partly the fault of the rules here which finds “read the article please” to be a distraction/nuisance but don’t mind that this often results in misinformed discussion propagating toward the top of a thread.
Just a heads up for the next guy...
DST varies by state in Australia. Western Australia (UTC+8) does not observe DST.
In this context, middle eastern countries with Islam as the majority religion have the day off on Fridays.
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
But doesn't Israel also not follow a typical work week? Eg. No commits on Friday afternoon?
…What I would really want is some input from Microsoft. I don't know what they can reveal, but I believe they must have quite a bit more data, than us. Even speaking about times, well, he must have been using Github more than to just commit stuff. And I'd imagine they have some logs. Also, it's very fair to assume than he was using VPN all the time, but it's also fair to assume that he wouldn't accidentally slip like that, revealing his real time zone in 9 commits. So, yeah, for sure there are people who have waay more data than me, and probably know how to use it properly better than me too. Not sure they will be willing to share, though.
I think I saw somewhere a claim that somebody confirmed he used a VPN, I'm can't recall how they did that.
I have a relative with mental health problems who lives on the street, and he sometimes writes me from public libraries -- kind of luckily for family members who are concerned about him, the webmail service he uses still includes that info in headers. So I usually know which public library he writes from.
i.e. most Israelis work those days.
Greece and Finland aren’t prolific. I’m sure they have the resources, many lone people would have. I don’t think they have the motives though. The US, Russia, China, NK and Israel have the motive, opportunity and track record. I wouldn’t rule out Ukraine either (and Ukraine is GMT+2)
I still think that an individual is the most likely candidate though.
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
But for any X which is mainstream today someone was always the first of X. And even then being the second or third is still cutting edge. For a certain kind of company it makes sense to play this way, but if it were everyone then we'd have a tragedy.
99.9% of releases for packages like this are boring bugfixes and stuff, not earth-shattering new features. You're not at a competitive disadvantage for not having taken this version of xz and having routed all of your traffic through it.
Seems to check out to me.
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
Me
> For a hacker, it is much more plausible to work in the afternoon and late at night
c/hacker/younger person
What is this syntax? I know s/a/b/ for sed substitution but not such a thing starting with c
It's the syntax for sed substitution for someone who mistypes, is dyslexic or does not have an excellent memory.
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
Oh, my sweet summer child.
Talk to me again in another decade or so.
Are you saying I will grow out of this and become a lark? Or are you saying I will eventually conform, oh wise one?
https://www.sleepfoundation.org/aging-and-sleep/why-do-older...
YMMV but the idea that you can't grow in/out of sleep patterns is empirically false, and I know this from personal experience.
For most of my life, I didn't need glasses for reading... until I suddenly did. Bodies change.
The article seems to be taking about people on the verge of entering "the facility", on their slow slide into dementia, who are making do despite not getting enough sleep. I'll get back to you when I hit 70.
I don't know how in the world you got that from the article, but no.
If we're putting stereotypes aside, wouldn't we also put aside the word "hackers"?
In the xz case, the term may be accurate, but we don't know the identity of the culprit(s) and have no empirical basis for even speculating about the work schedules of such people.
> If you're having peaceful mornings, I envy you.
Well, I don't work in an office. I work alone. Like a stereotypical "hacker".
This is an interesting take. Most software engineers I know love to show off their FOSS contributions. It's a place to do what _you_ want to do, free of budgets and MBA interference. I guess in certain parts of the world you probably want to keep that sort of skill set under wraps, though.
Edit: speaking of, these guys might also have some time pressure at the moment with regards to hacking stuff lol. Kind of fits the bill with what we know about the hack
If I would be pretending something I'd certainly put at least one more indirection in.
In such setup nothing bad happens in such a leak.
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
There was WAY more than one person involved here. Every email, commit, etc. was a deliberate task that was carefully planned.
https://www.mail-archive.com/xz-devel@tukaani.org/