FWIW: that whole frame is just wrong anyway. Microsoft breaks stuff constantly. Everyone out there (including anti-cheat vendors) is relying on hackery and trickery to make their stuff work on windows. And every windows release a ton of that stuff breaks and these companies drop everything to re-reverse-engineer whatever it is they need.
What works compatibly[1] is the win32 C API implemented along with the specific set of DLLs that you built your application against[2]. MS curates the kernel/driver interface below that fairly well[3]. But the kind of trickery EAC is doing here is right in the middle of things that absolutely do not work on windows either.
[1] In the sense of "do this and your app won't break".
[2] Which is why windows binaries ship their own C libraries. FWIW this model exists in Linux too. It's basically the way we package Docker and Snap containers. Had EAC been written against a particular Snap environment, we wouldn't be having this problem either because it would be frozen on a particular glibc version.
[3] And above that level, there are often other runtimes like .NET to which you can portably target.
I strongly favor the Linux\Linus "don't break userspace" attitude than what has just happened for glibc.
This is especially salient given that the DT_GNU_HASH format is not even documented. Leaving it as the only-option-by-default going forward is unconscionable.
> On typical distributions, none of the system libraries use DT_HASH, not even core libraries such as libgcc_s.so.1. Any tool that performs symbol lookups needs to support DT_GNU_HASH these days.
It seems to me that EAC includes its own copy of a dynamic linker for the express purpose of validating that glibc functions are not obviously detoured in the current process.
It's equally possible that they added DT_HASH lookup to achieve glibc 2.34 compatibility. The old way for getting the original dlsym function after interposing dlsym stopped working because the (internal, usually uninterposed, now unused) __libc_dlsym@@GLIBC_PRIVATE symbol went away in glibc 2.34. But a program can still obtain the dlsym function address by looking at the ELF data structures directly.
This would explain why people are so annoyed: there's unplanned extra low-level work to enable glibc upgrades two years in a row.
There is the strict interpretation of the words "what is an abi?" and then there is the real world "what breaks things users would rather not be broken?"
Maintainers of a package used by literally almost every executable really ought to use the latter thought process, IMHO. Hiding behind the "well, technically..." is kind of irresponsible. Hiding behind "It's been 16 years since we introduced $NEW_THING" is doubly so. Who the hell reads glibc's changelists for fun?
You know the quote: "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.`"
Sympathy for the problems for EAC seems to be in short supply, and I can sympathize. It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time. What I would expect to see would be something like
step 1: provide complete and accurate documentation for the new format and communicate it widely and prominently to the developer community
step 2: set a target date & version for removal, and get feedback from the community to ensure it gives sufficient time and information to complete a migration. Provide a way for developers to ask for and get support to make the necessary changes. Release sample code, if possible.
step 3: one release before removal, loudly note the deprecation and the upcoming change.
step 4: as the release that removes backward compatibility is being prepared, again, communicate the upcoming change and provide guidance for those who can't update on any ways to mitigate problems.
step 5: finally, release the version with the breaking changes, and put the break boldly at the top of the release notes/changelog. Reiterate the previous guidance on mitigating problems and help.
> It doesn't really excuse the abrupt removal of the compatible hash table format without at least a deprecation effort ahead of time.
I strongly suspect that the reason for the "abrupt" removal was that it wasn't expected that anything would break: that table is used for dynamic linking, and the dynamic linker comes with glibc itself, so it can be assumed to understand the new table (and the dynamic linker which comes with other C libraries would be used to load these C libraries, and not glibc since you cannot load more than one C library in a single process). AFAIK, the code in the dynamic linker which understands files with only the old table wasn't removed, so from that point of view, compatibility was being kept.
That is, they didn't expect anything to be using that table without going through dlopen() and dlsym().
Exactly. In addition to that, it seemed very likely that the glibc 2.34 reorganization already broke software which depended on such hidden implementation details.
If that were the case, the plan should've been to revert if things do break, i.e. revert an improper action when you find it's based on false premises. But that's not what we see here.
> since you cannot load more than one C library in a single process
You actually can, and there are even some benighted corner cases where you need to (eg, providing a fake glibc to a DRM/anticheat system or other such malware while your plugins use a different one). It's not properly supported and making it work is !!fun!! in the Dwarf Fortress sense.
Going through great pains to support EAC is a bit ridiculous. EAC itself is ridiculous from the get-go since it doesn't actually prevent cheaters it just pretends. It's not even meant to stop cheaters! Its entire purpose is to lock developers into Epic's proprietary system and prevent them from porting to non-approved platforms.
The entire premise of client side anti-cheat tools is, "trust the client." I think pretty much everyone at Hacker News knows that's the exact opposite of how you make a secure system.
They didn't break backwards compatibility, only forwards. The core issue here is that there is a new format that won't work with old runtime linkers. But the only way to present that format to those old linkers[1] is to take a binary compiled with a glibc version from the last few weeks (!) and run it against on a system where the ld-linux.so is from pre-2006 glibcs. That direction of compatibility is not impossible, but it's not what one normally considers when discussing "breaking backwards compatibility".
Fundamentally the only way to provide the kind of compatibility you're demanding, where any level of reverse engineering or parallel implementation must be supported in perpetuity, is to just Never Change Anything At All.
[1] With the sole known exception of the "runtime linker" embedded in this EAC product under discussion.
If the required symbol versions are provided by the old glibc version (which is a bit unlikely), it will still work. The internal glibc hash tables are ignored by the link editor, and most distributions override the link editor hash table default anyway. (What changed is that glibc no longer overrides the distribution defaults.)
Keeping a existing binaries working is the definition of backwards compatibility in an OS. glibc broke that.
Never Change Anything is far from the ask here. The ask is that when real actual breakage is found, the benefits of keeping said breakage justify not spending the engineering costs to fix backwards compatibility. Near as I can tell, the benefit is 16kB disk space savings, and glibc's engineering cost to maintain compatibility is nothing.
Like, another backwards compatibility breakage I saw recently was Graphene switching from 39-bit to 48-bit virtual address space, which also broke games. That has a real measurable security benefit, and actual engineering costs to develop a backwards compatibility story. But they still spent effort to fix breakages.
> Keeping a existing binaries working is the definition of backwards compatibility in an OS. glibc broke that.
No, that's simply wrong. No "glibc binary" that could have been even theoretically compiled within the past few decades will fail to run because of this issue.
What happened is that this EAC product had reverse-engineered the internal formats glibc uses to provide the APIs it implements. And that format changed.
I repeat what I said elsewhere: it is simply not possible for any software product to anticipate all possible external users of its internal implementation details. What you want, you cannot have. (And I repeat what I said elsewhere too: you don't have this, even on windows. Microsoft breaks things like anti-cheat hackery with every release too, it's just that you don't notice because the proprietary vendors care enough to fix it, where they don't on Linux.)
> No "glibc binary" that could have been even theoretically compiled within the past few decades will fail to run because of this issue.
That's obviously not true. The entire story is how a new glibc binary broke existing binaries.
> What happened is that this EAC product had reverse-engineered the internal formats glibc
Okay I missed this; I thought EAC was parsing the libc binary as per ELF. What was reverse engineered?
> it is simply not possible for any software product to anticipate all possible external users of its internal implementation details
Correct. Which is why I never asked for stuff that never breaks.
What is important is that when stuff is found to break, the breakage and refusal to provide backwards compatibility has a compelling reason. glibc's justifications here are laughable; they're spending more effort to not revert the breakage than it would take to fix.
Not trusting the client can only get you so far, in some types of game it's unavoidable that the client must have enough information and control that a cheat can enhance human play or replace the human altogether. It will always be possible to enhance aiming in a shooter, or automate repetitive tasks in an MMO, or script perfect blocks and parries in a fighting game, or feed the state of an online chess game into Stockfish and have it play for you, even if you trust the client to the minimum possible extent.
The only things that can be done in those cases are loosely "detecting" cheaters through heuristics and human review of suspicious behaviour, which won't catch more careful cheaters and may lead to false positives, and client side solutions like EAC which frustrate less skilled cheat developers and sometimes catch the more skilled ones off guard. Alternatively, give up on PC and focus on consoles which are sufficiently locked down that cheating is currently impossible regardless of how much you trust the client.
Or server-side automated review using machine learning models, which is what Valve is trying with "vacnet" (with then manual review to confirm the model's results are accurate - something most tech companies should learn from)
It's an interesting idea, but it's hard to say if it's really effective against anything besides blatant ragehacking. AFAICT the general consensus in the community is that Valves own CS:GO servers are still infested with cheaters, and to avoid them you need to go to ESEA or FACEIT (third party CS:GO matchmaking services) or Valorant (Riots CS:GO clone) which all primarily rely on very aggressive clientside scanning solutions.
I suspect for ESEA & FACEIT the most effective aspect of their anticheats is just that they are paid services. If I'm going to cheat at a game, I'm generally going to bias towards minimal consequences if I do get caught.
I can say that Valorant's anti cheat is the singular reason I've never tried it, even though I love counter-strike and similar games. Giving game devs (especially those owned by Tencent) ring 0 access is, uh, a step too far I think.
Valve took away the methods of getting into Prime Matchmaking without paying (you must pay $15 now) and it doesn't seem to have improved the situation much if at all.
The ship has sailed on ring0 really, all of the major anticheat players plug into ring0 on Windows. If you play online games on Windows there's almost no avoiding it.
Interestingly Proton acts as a loophole, EAC and BattlEye don't touch ring0 on Linux even though they do on Windows. Gamedevs have to opt-in to allowing Linux clients though.
> If steam deck is successful enough I wonder how long until kernel modules become a requirement...
Yeah, there's already Linux-specific cheats in the wild which abuse the fact that the Linux anticheats are oblivious to anything happening outside of the current user, so they can manipulate the games memory with complete impunity simply by doing it from root or a kernel module. Developers may become more hesitant to allow Linux clients as abuse becomes more common, unless the anticheats start extending into ring0 on Linux the same way they do on Windows.
Yeah. These things have to literally take over the machine in order to even have a chance at preventing anything. They're basically malware at this point. EAC is implemented as a kernel module, at least in Windows. It's bad enough to have to put up with proprietary nvidia drivers, now we're gonna have to tolerate this? It doesn't even work, people will still cheat.
I seriously hope these corporations don't start pushing this on Linux gamers. They need to accept that they can't control it and move on.
The issue with the server-client relationship in games is there a trade-off between latency and security required.
Offload too many tasks to the server and you end up with a game where you encounter latency where it's critical, like hit detection (and likely many others). Those extra milliseconds does make or break the experience.
Not saying it's an impossible problem to solve, just that multiplayer games are operating in a time-sensitive environment.
> Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on.
Most problems like this are communication and co-ordination problems.
Unless there’s bots out there scanning public code repos for esoteric stuff like this I kinda doubt it. If you mean it would have gotten caught by packagers then yeah but that presumes a world where `apt install eac` makes any sense.
We do actually have a way to rebuild Fedora with particular changes (eg. when we change the baseline x86-64 version). I guess other distros can do that too.
Of course it goes almost without saying that EAC would make about as much sense as an open source project as it does as a closed source project, which is to say none whatsoever because it's conceptually nonsense :-)
Neither does a closed source one, it is running on _my_ computer after all. Cynically.. keeping it closed just allows you to call it "anti-cheat" with a straight face, rather than what it actually is.. "mildly successful cheat mitigation and frustration techniques."
Which is obvious once you notice there is an active "anti-anti-cheat" marketplace.
That's a poor argument because, even going strictly by the specification, DT_HASH isn't authoritative--the symbol table is provided by DT_SYMTAB.
Personally, I feel that by the time you're messing with the _DYNAMIC entries of an ELF file, you need to be prepared to actually follow and keep up with ELF handling rules. Deleting DT_HASH feels no different than, say, changing the format of the structure pointed to by DT_DEBUG, which happened just last year (https://sourceware.org/bugzilla/show_bug.cgi?id=15971, https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a93d9e03...). Which, you know, breaks debuggers--but no one's running around saying "oh no glibc's ABI story is complete and utter trash!" because of that change (I daresay most of the people commenting on this issue aren't even aware that it happened).
If glibc updates its version number and EAC starts failing because it used too small a data type for that, does that mean glibc broke ABI compatibility and should be stuck on a lower version forever?
Closed source software that wants to hold back open source software can go pound sand. Gamers can run their games with older glibc in a chroot / container. I say this as someone who does exactly this for games.
In the Windows world the answer to that question was emphatically "Yes, you should contort yourself to make sure users can still actually USE their computer", to the point that the official version number of windows was "3.95" because some apps looked at only the minor version number and would have broken otherwise.
Other stuff like the game Shovel Knight and even an open source frame limiting library broke because of this change, there are probably more things but this has not reached the masses yet.
I dont think the change was inherently bad but just ham fisting it into the release without widely announcing that the ABI will change is a massive failure on the glibc teams end.
I don't know what people expect glibc to do here. EAC is a tool that reverse engineered library data that is only supposed to be used by the system runtime linker. People reverse engineer stuff all the time, and it's universally agreed that the onus is on the Weird Hack to bother with maintenance. EAC isn't using a supported API at all (the API for this is dlopen/dlsym et. al., not the hash format!).
It's unfortunate, sure, but it's just not possible to prevent. And all the people screaming about "This Doesn't Happen On Windows" are just fooling themselves (or, to be blunt, being deliberately disingenous). This happens on windows every single release. Windows N+1 comes out and suddently a bunch of hacky drivers or tools or gadgets stop working until their authors manage to figure out what part of their trickery Microsoft broke.
Anticheat software like EAC (as well as commercial antiviruses and malicious rootkits) often reverse-engineer internal system data structures and call undocumented functions. You really can't blame system library developers for breaking compatibility with software that is doing things "the wrong way."
Yes, I know Linus Torvalds says "don't break userspace," but that's for officially exposed kernel interfaces.
nothing was "reverse-engineered". Read the article. DT_HASH has been documented in SysV since ages. It's the DT_GNU_HASH that's (almost) undocumented and needs to be reverse-engineered from GNU ld's source code.
Seems like there is some handwaving here that "DT_HASH" should be supported forever because "The DT_HASH format is mandated by the System V ABI specification"
It was replaced by DT_GNU_HASH in the gnu implementation of libc
....I think these people have forgotten what GNU stands for.
Some time ago, https://refspecs.linuxfoundation.org/ was created to fix that problem. I don't know whether it's been kept up to date since then, however.
47 comments
[ 6.0 ms ] story [ 103 ms ] threadhttps://news.ycombinator.com/item?id=32471624 ("Win32 Is the Only Stable ABI on Linux?", over 500 comments)
What works compatibly[1] is the win32 C API implemented along with the specific set of DLLs that you built your application against[2]. MS curates the kernel/driver interface below that fairly well[3]. But the kind of trickery EAC is doing here is right in the middle of things that absolutely do not work on windows either.
[1] In the sense of "do this and your app won't break".
[2] Which is why windows binaries ship their own C libraries. FWIW this model exists in Linux too. It's basically the way we package Docker and Snap containers. Had EAC been written against a particular Snap environment, we wouldn't be having this problem either because it would be frozen on a particular glibc version.
[3] And above that level, there are often other runtimes like .NET to which you can portably target.
This is especially salient given that the DT_GNU_HASH format is not even documented. Leaving it as the only-option-by-default going forward is unconscionable.
It seems to me that EAC includes its own copy of a dynamic linker for the express purpose of validating that glibc functions are not obviously detoured in the current process.
This would explain why people are so annoyed: there's unplanned extra low-level work to enable glibc upgrades two years in a row.
Maintainers of a package used by literally almost every executable really ought to use the latter thought process, IMHO. Hiding behind the "well, technically..." is kind of irresponsible. Hiding behind "It's been 16 years since we introduced $NEW_THING" is doubly so. Who the hell reads glibc's changelists for fun?
You know the quote: "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.`"
step 1: provide complete and accurate documentation for the new format and communicate it widely and prominently to the developer community
step 2: set a target date & version for removal, and get feedback from the community to ensure it gives sufficient time and information to complete a migration. Provide a way for developers to ask for and get support to make the necessary changes. Release sample code, if possible.
step 3: one release before removal, loudly note the deprecation and the upcoming change.
step 4: as the release that removes backward compatibility is being prepared, again, communicate the upcoming change and provide guidance for those who can't update on any ways to mitigate problems.
step 5: finally, release the version with the breaking changes, and put the break boldly at the top of the release notes/changelog. Reiterate the previous guidance on mitigating problems and help.
step 6: Go have a tasty beverage.
I strongly suspect that the reason for the "abrupt" removal was that it wasn't expected that anything would break: that table is used for dynamic linking, and the dynamic linker comes with glibc itself, so it can be assumed to understand the new table (and the dynamic linker which comes with other C libraries would be used to load these C libraries, and not glibc since you cannot load more than one C library in a single process). AFAIK, the code in the dynamic linker which understands files with only the old table wasn't removed, so from that point of view, compatibility was being kept.
That is, they didn't expect anything to be using that table without going through dlopen() and dlsym().
You actually can, and there are even some benighted corner cases where you need to (eg, providing a fake glibc to a DRM/anticheat system or other such malware while your plugins use a different one). It's not properly supported and making it work is !!fun!! in the Dwarf Fortress sense.
The entire premise of client side anti-cheat tools is, "trust the client." I think pretty much everyone at Hacker News knows that's the exact opposite of how you make a secure system.
Never trust the client.
So at this point, the maintenance burden on glibc is however much effort they want to spend justifying breaking backwards compatibility.
Fundamentally the only way to provide the kind of compatibility you're demanding, where any level of reverse engineering or parallel implementation must be supported in perpetuity, is to just Never Change Anything At All.
[1] With the sole known exception of the "runtime linker" embedded in this EAC product under discussion.
Never Change Anything is far from the ask here. The ask is that when real actual breakage is found, the benefits of keeping said breakage justify not spending the engineering costs to fix backwards compatibility. Near as I can tell, the benefit is 16kB disk space savings, and glibc's engineering cost to maintain compatibility is nothing.
Like, another backwards compatibility breakage I saw recently was Graphene switching from 39-bit to 48-bit virtual address space, which also broke games. That has a real measurable security benefit, and actual engineering costs to develop a backwards compatibility story. But they still spent effort to fix breakages.
No, that's simply wrong. No "glibc binary" that could have been even theoretically compiled within the past few decades will fail to run because of this issue.
What happened is that this EAC product had reverse-engineered the internal formats glibc uses to provide the APIs it implements. And that format changed.
I repeat what I said elsewhere: it is simply not possible for any software product to anticipate all possible external users of its internal implementation details. What you want, you cannot have. (And I repeat what I said elsewhere too: you don't have this, even on windows. Microsoft breaks things like anti-cheat hackery with every release too, it's just that you don't notice because the proprietary vendors care enough to fix it, where they don't on Linux.)
That's obviously not true. The entire story is how a new glibc binary broke existing binaries.
> What happened is that this EAC product had reverse-engineered the internal formats glibc
Okay I missed this; I thought EAC was parsing the libc binary as per ELF. What was reverse engineered?
> it is simply not possible for any software product to anticipate all possible external users of its internal implementation details
Correct. Which is why I never asked for stuff that never breaks.
What is important is that when stuff is found to break, the breakage and refusal to provide backwards compatibility has a compelling reason. glibc's justifications here are laughable; they're spending more effort to not revert the breakage than it would take to fix.
Which is kinda why the story blew up.
No, you got it the wrong way around. EAC has used the well-documented and SysV-standard DT_HASH mechanism.
The DT_GNU_HASH is the (almost) undocumented one that they would have to reverse-engineer from GNU ld's source code
The only things that can be done in those cases are loosely "detecting" cheaters through heuristics and human review of suspicious behaviour, which won't catch more careful cheaters and may lead to false positives, and client side solutions like EAC which frustrate less skilled cheat developers and sometimes catch the more skilled ones off guard. Alternatively, give up on PC and focus on consoles which are sufficiently locked down that cheating is currently impossible regardless of how much you trust the client.
Great talk on it here: https://www.youtube.com/watch?v=kTiP0zKF9bc
I can say that Valorant's anti cheat is the singular reason I've never tried it, even though I love counter-strike and similar games. Giving game devs (especially those owned by Tencent) ring 0 access is, uh, a step too far I think.
The ship has sailed on ring0 really, all of the major anticheat players plug into ring0 on Windows. If you play online games on Windows there's almost no avoiding it.
Interestingly Proton acts as a loophole, EAC and BattlEye don't touch ring0 on Linux even though they do on Windows. Gamedevs have to opt-in to allowing Linux clients though.
If steam deck is successful enough I wonder how long until kernel modules become a requirement...
Yeah, there's already Linux-specific cheats in the wild which abuse the fact that the Linux anticheats are oblivious to anything happening outside of the current user, so they can manipulate the games memory with complete impunity simply by doing it from root or a kernel module. Developers may become more hesitant to allow Linux clients as abuse becomes more common, unless the anticheats start extending into ring0 on Linux the same way they do on Windows.
I seriously hope these corporations don't start pushing this on Linux gamers. They need to accept that they can't control it and move on.
Offload too many tasks to the server and you end up with a game where you encounter latency where it's critical, like hit detection (and likely many others). Those extra milliseconds does make or break the experience.
Not saying it's an impossible problem to solve, just that multiplayer games are operating in a time-sensitive environment.
> Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on.
Most problems like this are communication and co-ordination problems.
Which only exist in this case because the software isn't open source. This would have been spotted years ago in open source.
Of course it goes almost without saying that EAC would make about as much sense as an open source project as it does as a closed source project, which is to say none whatsoever because it's conceptually nonsense :-)
Which is obvious once you notice there is an active "anti-anti-cheat" marketplace.
Personally, I feel that by the time you're messing with the _DYNAMIC entries of an ELF file, you need to be prepared to actually follow and keep up with ELF handling rules. Deleting DT_HASH feels no different than, say, changing the format of the structure pointed to by DT_DEBUG, which happened just last year (https://sourceware.org/bugzilla/show_bug.cgi?id=15971, https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a93d9e03...). Which, you know, breaks debuggers--but no one's running around saying "oh no glibc's ABI story is complete and utter trash!" because of that change (I daresay most of the people commenting on this issue aren't even aware that it happened).
Closed source software that wants to hold back open source software can go pound sand. Gamers can run their games with older glibc in a chroot / container. I say this as someone who does exactly this for games.
BTW my comment lists two ways for people to continue "still actually USING their computers".
I dont think the change was inherently bad but just ham fisting it into the release without widely announcing that the ABI will change is a massive failure on the glibc teams end.
It's unfortunate, sure, but it's just not possible to prevent. And all the people screaming about "This Doesn't Happen On Windows" are just fooling themselves (or, to be blunt, being deliberately disingenous). This happens on windows every single release. Windows N+1 comes out and suddently a bunch of hacky drivers or tools or gadgets stop working until their authors manage to figure out what part of their trickery Microsoft broke.
Yes, I know Linus Torvalds says "don't break userspace," but that's for officially exposed kernel interfaces.
It was replaced by DT_GNU_HASH in the gnu implementation of libc
....I think these people have forgotten what GNU stands for.