> The normal way I use machines these days is that anyone with login access has some ability to become root.
That is good, but you don't necessarily need login access to exploit this. You could exploit it via an RCE vulnerability in some other program, like the HTTP server or NTP client.
I'll bet you don't. The natural extension of that logic is that you don't bother with UID separation for running processes (since you don't believe it provides any protection), so for convenience all your software runs as root and all your ssh logins are directly to root, etc... No one does that, and I'll bet anything you don't either.
In fact the Unix uid/gid does provide real security value in practice. It is not absolute, and relative to other barriers commonly used (LSM, cgroup containers, VMs, firewalls, BPF syscall filtering, etc...) it's comparatively weak and porous. But it's not something to dismiss either.
The nature of "defense in depth" security analysis is that you have to embrace that depth and not throw bits out. UID separation has the distinct advantage of being dirt simple to understand and extremely cheap to deploy. Use it.
>> The normal way I use machines these days is that anyone with login access has some ability to become root.
> I'll bet you don't
I'll... bet they do? You realize that description most trivially matches "I give people sudo access", right?
So they say they do something
and you promptly say that you don't think they do that, because "nobody" does a more extreme version of the same idea that they never suggested? That's a classic straw man.
That seems like a maximally uncharitable way of interpreting what I thought was a pretty uncontroversial point. The point was that when deploying software for use, best practice says "don't run it as root", and not "assume all software has root access". Do you really disagree? Because the grandparent poster definitely seems to, and I think that's pretty clearly wrong.
That still seems senselessly combative. I wasn't being mean, at least re-reading my comment it doesn't seem so to me. What's the deal here? Who did I offend, and how?
To repeat a third time: don't run stuff as root simply because you assume root elevation exploits exist. I stand by that, and don't really think anyone here disagrees.
I don't think you were being mean, but you are being dense. This is the OP's statement:
> anyone with login access has some ability to become root
If you take that to mean "I run all my software as root", then you're being maximally uncharitable. And when people point out to you that your interpretation isn't the only one available (nor the most logical one), you accuse others of being senselessly combative?
> I wasn't being mean, at least re-reading my comment it doesn't seem so to me.
I don't have a dog in this race, but you started your comment with "I bet you don't", which, to me, is unnecessarily combative and/or snarky. Right out of the gate I was predisposed to think ill of whatever you had to say.
FWIW: the grandparent has editted the comment to remove the framing I was responding to (something along the lines of "root access is inevitable"), which explains why everyone is being so outrageously touchy.
I guess I should have quoted. My apologies for seeming "snarky", but the conversation you read isn't the one I was engaged in.
But let me repeat for a fourth time that you shouldn't be running processes as root, whether or not you have worries about undiscovered vulnerabilities and especially regardless of whether you are "predisposed to think ill" of the people telling you that.
UID separation isn't a perfect model, but it's a good one, and you should use it.
Quite disappointing someone would enable that kernel option without fully considering the ramifications :/.
Do some distributions use systemd-coredump by default? I not just learned about systemd-coredump and I'm wondering if it could be actually useful for develoment purposes; apport doesn't seem to be.
I use systemd-coredump all the time on Fedora, it's pretty convenient. When a program crashes, run "coredumpctl gdb" to launch gdb on the latest coredump.
systemd-coredump is vuln, systemd-oomd kills the wrong stuff, and you don't even learn that these programs exist (and are running on your system) until you find out from a CVE or a blog post detailing how to mitigate the problem[1].
And then, there are very few alternatives, because more and more software requires systemd (and even if most things still don't, so much stuff requires glibc so Alpine or OpenBSD aren't always very good as alternatives).
You can turn the situation around: Pick some non-systemd OS, stick to it, and avoid software that does not run on it. You will end up with the old and boring tech, but it will be robust and run more reliable in the long term.
I've been running Alpine Linux in production for 4 years now.
Not always. I have sway and pipewire running smooth as can be on my systemd-free workstation. As well as recent releases of Linux, glibc, Mesa, and Firefox. All the essentials are there and up to date for me!
Unfortunately systemd & glibc (or even macOS) are usually the more pragmatic choice. I'd love to have the freedom to ignore them, but even outside of work, there is software (like games) that I want to run that doesn't play nice with anything more sane.
The glibc 2.26 breakages (including the ucontext thing) were what made me stick to musl/Alpine. This was in 2017, around the same time poettering did the "i don't consider this much of a problem"-things on github.
Alpine's great. My entire homelab runs on it. Only complaint I have is its IPv6 support sucks. I have to hack together a bunch of scripts to get a reasonable network configuration.
Note that musl libc in Alpine Linux has significantly smaller default thread stack size than glibc (128 kB vs 8 MB), therefore software developed and tested with glibc can easily crash with stack overflow.
systemd-coredump is honestly incredibly useful. Any time an application randomly crashes in the background, you can easily pop it in a debugger to look through the backtrace without having to re-run it. Very handy for when the crashes are not easily reproducible, or you didn't expect something to fail and are not sure how to make it fail again.
As far as I can tell all the magic you're describing is provided by the kernel's core dumper and not systemd. You can achieve the same effect by creating /tmp/cores and setting a meaningful pattern in /proc/sys/kerne/core_pattern.
It's also incredibly annoying, because it changes longstanding cross-platform common behavior regarding this exact functionality. "A coredump is written to the crashing process's current working directory, named 'core' or 'core.<pid>'" - that used to be true on Linux and all the BSDs.
Several setups of ours rely on that fact, and since we're developing software that is supposed to work on Linux and {Free,Open,Net}BSD, we were happy to be able to reuse a bunch of crash-capture logic.
systemd kneecapped us out of the blue, it took us some time to figure out where the coredumps were going. Now we disable systemd-coredump on all systems. Which works, except when something reenables it or we add a new platform and someone forgets to disable it. Sigh.
Yes, we can deal with it, by disabling it. But crash captures went from being reliable to "did systemd-coredump creep back in?" - a definite regression.
That default unix way is very annoying. I want all my core dumps collected in one place so I can easily monitor for new core dumps and easily find them. I do not want to have too search for them. The tool does not have to be systemd-coredump but it is good enough for the job.
Well, the used-to-be-true logic also resulted in scripts to find where there were core files, to remove them :).
At least in Debian and Ubuntu systemd-coredump is a separate package (and wasn't installed by default for me atleast), so I expect you can just remove it and it doesn't get automatically enabled in any way.
It's not like it was ever a safe assumption to blindly dump core to a process' CWD. It was just a convenient option in a developer-centric UNIX mindset, in lieu of anything better.
Sometimes daemons run with a CWD changed somewhere rarely if ever written to at runtime, and presumed to primarily contain information intended for sharing. It's not hard to imagine a security-conscious daemon that's done both chroot() and chdir() into a path intended for publishing information out of, and not generally intended for the daemon to write into except by privileged users/publishers.
When that daemon spuriously dumps core into its CWD now you have potentially sensitive information sitting in the published tree, e.g. maybe the credentials of the privileged users are in that core file.
I really don’t get why people are so quick to jump on “it changed so it’s bad.” systemd-coredump has to work in every possible case and is dealing with every unhappy path you can imagine. The kernel has a laundry list of situations where core files can’t be saved and systemd made the choice to set it up so some of the failure modes can’t happen.
If they just tried to write to cwd there would be a blogpost about “systemd loses coredumps if your app directory is read-only” which isn’t some theoretical problem, it’s containers.
> I really don’t get why people are so quick to jump on “it changed so it’s bad.”
That's not actually the issue here. Rather: "it changed so now it's inconsistent across platforms"
> There was no way to really win here.
There is:
- if you're changing a multi-platform standard, include the other platforms (e.g. create a coredumpctl tool for *BSD)
- do the changes slowly/incrementally in a backwards-compatible way (e.g. apply the new coredump locations first to read-only containers and service daemons running as systemd units)
> systemd-coredump is honestly incredibly useful. Any time an application randomly crashes in the background, you can easily pop it in a debugger to look through the backtrace without having to re-run it.
coredumps are of course incredibly useful, as they have been for decades. Couldn't live without them.
Is systemd adding any value here other than making things more complex and brittle?
The exploit is quite interesting. Initially I didn't believe there could be a problem -- after all, we have the source of "su" so who cares if we can see a coredump from it? However the exploit is that "su" reads normally hidden files like /etc/shadow into memory (or more likely PAM does) thus exposing the contents of these files like password hashes to the non-root user.
I wonder if it would be generally better for su/PAM to mark this data as non-dumpable (MADV_DONTDUMP), just for defence in depth. You could also imagine coredumps being uploaded in bug reports, unintentionally revealing sensitive data.
This reminds me of how several years ago Microsoft was found to be uploading mini dumps for the crash reporting service ... Using plaintext http. Probably fixed by now.
When will RedHat learn that too many knobs and buttons, especially global settings, result in leaks and exploits? I don't care if the customers demand these.
I'm quite happy to enable/disable core dumps in the shell. Make Linux great again!
Obligatory comment that the only way to make a stable and well understood Linux system is to uninstall systemd, and shame on any distro that forces you to use this shambling corpse of a project.
51 comments
[ 3.0 ms ] story [ 117 ms ] threadBut anyone who is hosting shellboxes (including "bastion" hosts in ISO27001 environments): this is pretty serious.
That is good, but you don't necessarily need login access to exploit this. You could exploit it via an RCE vulnerability in some other program, like the HTTP server or NTP client.
In fact the Unix uid/gid does provide real security value in practice. It is not absolute, and relative to other barriers commonly used (LSM, cgroup containers, VMs, firewalls, BPF syscall filtering, etc...) it's comparatively weak and porous. But it's not something to dismiss either.
The nature of "defense in depth" security analysis is that you have to embrace that depth and not throw bits out. UID separation has the distinct advantage of being dirt simple to understand and extremely cheap to deploy. Use it.
> I'll bet you don't
I'll... bet they do? You realize that description most trivially matches "I give people sudo access", right?
So they say they do something and you promptly say that you don't think they do that, because "nobody" does a more extreme version of the same idea that they never suggested? That's a classic straw man.
To repeat a third time: don't run stuff as root simply because you assume root elevation exploits exist. I stand by that, and don't really think anyone here disagrees.
> anyone with login access has some ability to become root
If you take that to mean "I run all my software as root", then you're being maximally uncharitable. And when people point out to you that your interpretation isn't the only one available (nor the most logical one), you accuse others of being senselessly combative?
I don't have a dog in this race, but you started your comment with "I bet you don't", which, to me, is unnecessarily combative and/or snarky. Right out of the gate I was predisposed to think ill of whatever you had to say.
I guess I should have quoted. My apologies for seeming "snarky", but the conversation you read isn't the one I was engaged in.
But let me repeat for a fourth time that you shouldn't be running processes as root, whether or not you have worries about undiscovered vulnerabilities and especially regardless of whether you are "predisposed to think ill" of the people telling you that.
UID separation isn't a perfect model, but it's a good one, and you should use it.
Do some distributions use systemd-coredump by default? I not just learned about systemd-coredump and I'm wondering if it could be actually useful for develoment purposes; apport doesn't seem to be.
[1]: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
And then, there are very few alternatives, because more and more software requires systemd (and even if most things still don't, so much stuff requires glibc so Alpine or OpenBSD aren't always very good as alternatives).
https://news.ycombinator.com/item?id=33894469 (2 weeks ago, 134 comments)
I've been running Alpine Linux in production for 4 years now.
Not always. I have sway and pipewire running smooth as can be on my systemd-free workstation. As well as recent releases of Linux, glibc, Mesa, and Firefox. All the essentials are there and up to date for me!
I am using Gentoo with no systemd and with the latest versions of all programs that I am interested in.
Several setups of ours rely on that fact, and since we're developing software that is supposed to work on Linux and {Free,Open,Net}BSD, we were happy to be able to reuse a bunch of crash-capture logic.
systemd kneecapped us out of the blue, it took us some time to figure out where the coredumps were going. Now we disable systemd-coredump on all systems. Which works, except when something reenables it or we add a new platform and someone forgets to disable it. Sigh.
Yes, we can deal with it, by disabling it. But crash captures went from being reliable to "did systemd-coredump creep back in?" - a definite regression.
Have you tried masking it? (https://www.freedesktop.org/software/systemd/man/systemctl.h...)
At least in Debian and Ubuntu systemd-coredump is a separate package (and wasn't installed by default for me atleast), so I expect you can just remove it and it doesn't get automatically enabled in any way.
Sometimes daemons run with a CWD changed somewhere rarely if ever written to at runtime, and presumed to primarily contain information intended for sharing. It's not hard to imagine a security-conscious daemon that's done both chroot() and chdir() into a path intended for publishing information out of, and not generally intended for the daemon to write into except by privileged users/publishers.
When that daemon spuriously dumps core into its CWD now you have potentially sensitive information sitting in the published tree, e.g. maybe the credentials of the privileged users are in that core file.
If they just tried to write to cwd there would be a blogpost about “systemd loses coredumps if your app directory is read-only” which isn’t some theoretical problem, it’s containers.
There was no way to really win here.
That's not actually the issue here. Rather: "it changed so now it's inconsistent across platforms"
> There was no way to really win here.
There is:
- if you're changing a multi-platform standard, include the other platforms (e.g. create a coredumpctl tool for *BSD)
- do the changes slowly/incrementally in a backwards-compatible way (e.g. apply the new coredump locations first to read-only containers and service daemons running as systemd units)
It isn't as if it worked the same everywhere anyway.
- The BSD are irrelevant. Why hamper Linux development to cater for users who don't exist?
coredumps are of course incredibly useful, as they have been for decades. Couldn't live without them.
Is systemd adding any value here other than making things more complex and brittle?
I wonder if it would be generally better for su/PAM to mark this data as non-dumpable (MADV_DONTDUMP), just for defence in depth. You could also imagine coredumps being uploaded in bug reports, unintentionally revealing sensitive data.
I'm quite happy to enable/disable core dumps in the shell. Make Linux great again!
How it's going: CVE-2022-4415