The separation made more sense when disk space was scarce, but I guess no one today is using separate partitions for /bin /sbin and /usr, or mounting /usr over NFS.
More granular filesystems limits the blast radius of filesystem corrupting events.
Hard disks used to fail pretty often, filesystems had (and still have) bugs. It makes sense to stick your administrative utilities needed to run/recover a given system in its own little less frequently modified filesystem vs. user applications that tend to see more churn/use in general.
Indeed most Linux distros today first boot into an initrd filesystem, mount the “real” rootfs somewhere, and then `pivot_root` to the real `/` while calling init (usually systemd) these days
An important part of passing the red hat certified system administrator (rhcsa) was to be able to recover a rhel system that was failing at booting… i got that in my exam, and i suspect that’s present in every exam session.
NixOS itself still also has problems. The configuration / state for a program is scattered between /etc /nix/store ~ ~/.config /var instead of being in a centralized place.
An application can be thought of as a combination of the binary, libraries, resources, and state. By default the applications needs to be able to read any of these and be able to read and write the state. Unix's FSH's biggest problem is that it tries to group files together by type instead of grouping files together by application. Nix can technically group things by application, but it is not really standardized so things can get messy and it is still normal for applications to read and write files from anywhere.
> NixOS itself still also has problems. The configuration / state for a program is scattered between /etc /nix/store ~ ~/.config /var instead of being in a centralized place.
Those are features, not bugs... There are reasons for all of those existing and for why you wouldn't want everything in a centralized place.
The same could be said about having /bin /lib /usr/bin /usr/sbin /usr/lib /usr/local/bin /usr/local/lib ... yet these were all merged into /nix/store. NixOS's handling of state is poor. Even ignoring the problem of it all being spread out it is not uncommon for secrets to make it into /nix/store which is world readable.
Secrets handling is indeed an issue, you can shoot yourself in the foot if you are not careful. If you are careful you can avoid it though and it is not neccessarily difficult. It is certainly is one aspect of NixOS that requires care and can be improved. But to me, it is not so bad as to avoid it. The advantages outweigh the disadvantages with a wide margin
Well, good luck convincing anyone to merge all config and mutable state into the same place (that should be fun, especially the user-specific config/state) :)
> NixOS's handling of state is poor. Even ignoring the problem of it all being spread out it is not uncommon for secrets to make it into /nix/store which is world readable.
The secrets handling is a well-known issue and easy to avoid when you're familiar with Nix. Although yes, it can be a problem when people are not aware of it.
And I disagree with NixOS's handling of state being poor. Apart from this "secrets" problem you mentioned, it's not worse than other Linux-based operating systems and in fact it can be argued that it's significantly better.
Just the fact that /nix/store is mounted read-only is, on its own, already a huge improvement over letting users and applications installing and modifying the system in an ad-hoc fashion (good luck when you upgrade your system!).
Not to mention all the other advantages of /nix/store (such as user-installed applications, no package or library conflicts even if you install multiple versions of them, etc).
And as another example, the `stateVersion` feature is something that also makes upgrades a lot more reliable, and I know of no other operating system that has a similar feature.
Atomic configuration changes, booting into specific configurations, and system-wide configuration roll-backs (configurations which also include changes in package versions or even entire OS upgrades), also makes upgrades and configuration changes easy to try / fix if anything goes wrong. No other widely-used operating system has such a reliable configuration change mechanism, as far as I know.
But, you know, if you have other ideas of how things can be improved (apart from your idea of merging user and system-wide state into the same place, which I think makes no sense), I would definitely appreciate to hear it!
If you can think of a concrete and detailed, workable proposal, maybe you could write up a NixOS RFC and send a PR for discussion (or post something on the discourse community website?).
Who knows, maybe something can be fleshed out and improvements come out of it...
Thanks for the suggestion, ut I don't currently have time to design this. If I did I personally think there are other usability issues that I would prioritized fixing instead of trying to push the technology forward.
> The configuration / state for a program is scattered between /etc /nix/store ~ ~/.config /var
That's not really scattered. /nix/store has the generated files. They get linked to /etc if the app requires that kind of thing - often it's just added to the arguments instead. ~/.config is for user preferences (possible to flip at app runtime or static) which are not a system setting. /var is for system services states. Those are very separate use cases and get managed in different ways. The apps should also never be aware that /nix/store is a thing.
>Those are very separate use cases and get managed in different ways.
To me they all look the same. For example you could have a /nix/state that had a directory for each program. Each program could have its own global and user specific state it could manage from within its directory.
Then you get firejail for more custom apps. You still need that layer and needs to be configured per-app. Would moving config+state into one path really improve the experience? In what way would it be simpler?
> For example you could have a /nix/state that had a directory for each program.
NixOS already has that. It's called `/var/lib`.
> Each program could have its own global and user specific state it could manage from within its directory.
The user-specific state is (and should be) in each user's home directory, not scattered around in a bunch of system-wide directories, one for each application...
That's not just for convenience, but also for security (and privacy) reasons.
I'd be interested to know who gets to manage the user-specific state in your proposal. Does each app have it's own per-user directory for each user? i.e. if you have 1000 apps and 10 users, do you need exactly 10,000 user-specific directories? And if not, who gets to create these directories and when are they created/removed?
To be honest, it sounds like you're not aware of all the complexities and implications / consequences of what you're proposing, i.e. it sounds like you didn't think this through. But on the off-chance I'm wrong, I'd be happy to be proven so.
>The user-specific state is (and should be) in each user's home directory, not scattered around in a bunch of system-wide directories, one for each application...
I disagree with this because state is owned by the program so it would make the most sense for it to live in the directory for the program. A user can tell the program that they want to modify the state, but ultimately the program has total ownership of its state. The program should not be leaching into user's home directory to read and write files and should be restricted to directories it owns.
>I'd be interested to know who gets to manage the user-specific state in your proposal.
One way to do it would be for the program to manage it.
>Does each app have it's own per-user directory for each user?
It could choose.
>if you have 1000 apps and 10 users, do you need exactly 10,000 user-specific directories?
This is unrealistic. On the want majority of computers an app will only be run by 1 user. Focusing on handling multiple users is a more special case and is a distraction from the point I'm trying to make.
> One way to do it would be for the program to manage it.
Really? So a program can access (and modify!) the state of all of its users?
Do you realize how easy it would be to trick my Firefox instance into reading (and even modifying) the state of another user, and even running other code on behalf of another user, given that it would have privileges to do so?
If I understand correctly your proposal, you basically just converted almost every single application bug into a privilege escalation bug.
Not to mention that how do you even define what the same "app" is?
Is Firefox 28 the same app as Firefox 45? And if so, can a user install his own Firefox (like you currently can on NixOS and even other OSes)?
Also, how would you backup all of your own per-user state, given that every app can decide to manage it differently (i.e. storing it in different files and/or directories inside their own per-app directory, or perhaps even in a single per-app database which you wouldn't have access to).
Look, I'm not trying to grill you, I just think you haven't thought this through (but again, I'm happy to be proven wrong).
> This is unrealistic. On the want majority of computers an app will only be run by 1 user. Focusing on handling multiple users is a more special case and is a distraction from the point I'm trying to make.
What does this even mean? We're talking about user-specific state. So there's different state for each user, which means there are multiple users, by definition!
So how is focusing on multiple users a distraction? It's the whole point...
As I mentioned, on NixOS, the system-wide mutable state is already stored on /var/lib/<app>, which is what you are (redundantly) proposing. So we're just arguing about how to handle user-specific state, nothing else.
> The same binary can be run by multiple users and the user running it affects what files it can read and write.
What you're proposing is not even possible with standard Unix permissions (without leaving a huge security hole). You're suggesting that each program can create the state for each user inside the program's directory but somehow cannot read or modify the state for other users.
This suggests the program runs with each user's privileges and therefore these directories would require the sticky bit for state creation (like /tmp does) but this would not prevent a user from roguely creating the state for another user (this is why files in /tmp should be created with random names, they should not be predictable).
Or maybe you're suggesting that all apps should run with setuid privileges (so that they can create the state for each user, but users themselves couldn't) but we already know that would be absolutely terrible for security.
So something like AppArmor (which AFAIK has security flaws) or even SELinux would be required to implement this scheme, I believe, if it's possible at all.
SELinux, AppArmor and other jail-like mechanisms can be a huge burden to manage, especially since most apps would require a substantial number of exceptions to a default security policy (think GUI apps, or apps that require access to a user's files, for example)...
Since most apps do require read/write access to a user's files, or permission to show things on a screen, I'm not sure if you're gaining much security with your scheme. There would be almost infinite ways for an app to exfiltrate or modify a user's files or trick the user into doing things on behalf of the app, I think (like running other apps...).
And if you're implementing a restrictive security policy with something like SELinux or even some kind of container/jail, you might as well do the same but without having to change each and every app to use the weird per-app directory containing the state of all users. This would save you a huge amount of work and potential security vulnerabilities, since private home directories are already secure by default (in terms of protecting a user from other users) and restricting an app not to access the state of other apps could be equally done when using home directories.
> The user has an idea on what the same app is. Unfortunately Nix does not properly understand this concept.
So a user can decide what "Firefox" is? Then what prevents a user from installing a rogue Firefox that exfiltrates the other users' state, in your scheme?
> That is one place. Some programs don't use it. It's not a centralized place of state for every program.
It is for programs that run system-wide and they should be using it.
Which system-wide programs are not using /var/lib for their mutable state?
It's not like they have many other places to save their state... I mean, there is /var/cache but that's for state that can be removed without data loss.
In NixOS, programs are installed in many different unique paths that are added to PATH through symlinks. There are very few programs added to static directories such as /usr/bin. This in itself may not sound like an improvement, but NixOS as a whole has a very different idea how to manage dependencies. Difficult to explain well, but it feels like fresh air compared to the slow evolution of historical Unix principles that most other Linux distributions follow.
It is called the Nix store, it is not a database as much as a special type of file hierarchy.
It is not horrible, but it breaks tradition for sure. The benefits are best understood when evaluating NixOS as a whole, of which the file hierarchy is just a part.
One of the main benefits is that you can have multiple versions of something installed in the store, with individual environments pulling in the specific version that they require. Makes it really easy (or at least possible) to keep old software running, or to keep multiple environments running on the same computer even though the people who use them don't cooperate.
The paths to things in the nix store have hashes in them, so they are not intuitive at all (and they’re not intended to be). Inside an environment, a program like Perl could still be located at /use/bin/perl, but it will be a symbolic link to the nix store. Alternatively, your environment might have a PATH variable that mentions all the necessary nix store paths. It depends on how you built that environment, which will depend on what the programs you run in that environment require.
On the other hand, your personal files I your home directory are left alone; they don't get put in the nix store. An environment that needs access to your personal files (such as your desktop) simply has your home directory mounted into it.
> In NixOS however /lib and /usr/lib do not exist. Instead all system libraries, binaries, kernels, firmware and configuration files are placed in the Nix store. The files and directories in /nix/store are named by hashes of the information describing the built data. All of the files and directories placed in the Nix store are immutable. /bin and /usr/bin are almost absent: they contain only /bin/sh and /usr/bin/env respectively, to provide minimal compatibility with existing scripts using shebang lines. User-level environments are implemented using a large number of symbolic links to all required packages and auxiliary files. These environments are called profiles and are stored in /nix/var/nix/profiles, each user having their own profiles. Structuring the system in this way is how NixOS obtains its key advantages over conventional Linux distributions, such as atomicity and rollback support.
I have never heard about GoboLinux. The linked page gives me the impression that GoboLinix tries to improve the FHS by using more self-describing directory names? But still follow the general principles of splitting a program's dependencies into a statically defined directory structure?
What was ever the point of the $PREFIX/bin vs $PREFIX/sbin division, anyway?
I understand the historical reasoning for the / vs /usr vs /usr/local split —
• / was a mount of the machine vendor's installation-engineering-provided bring-up tape — a tape that could usually be safely unmounted once boot was complete and the kernel + persistent daemons had loaded into RAM. (We still have something akin to this in Linux land in the form of the initramfs!)
• /usr was a read-only-mounted OS distribution, either spooled onto the same tape drive that had been reading the rootfs tape a moment before, or (later) over NFS from a high-read-IOPS shared disk server, where in either case the mountpoint was expected to never go away while the OS was running, and therefore the loader could spawn binaries from here using flushable memory-mapped disk IO (allowing far more running processes to fit within the machine's limited memory — important for time-sharing!);
• /usr/local was a separate site-specific mount of stuff added by the local sysadmin — often created as a separate "tacked on the end" partition on the OS distribution tape (you could just attach more tape on the end of the reel, and then format it!) Later, once machine-local hard disks were a thing but Unix systems were still generally mounting /usr over NFS, /usr/local evolved to be the read-only partition of the machine-local HD (and such systems gained a matching read-write local-disk partition, /var.)
But $PREFIX/bin and $PREFIX/sbin were, as far as I know, always just regular directories under the same $PREFIX-level mountpoint, never separate mountpoints of their own.
The only difference I can even think of between a $PREFIX/bin and a $PREFIX/sbin dir, is that the $PREFIX/sbin dir wasn't in $PATH for most users. But surely that was never a real line of defense for security?
System V introduced /usr/sbin for storing loosely defined "system binaries", meaning admin tools, daemons and such like. The idea was that they wouldn't appear on the ordinary user's $PATH (and confuse them, I suppose).
Eh, afaik /usr/local was for storing stuff installed by the local sysadmin and not distributed by the vendor… ~/.local the was for stuff compiled by each individual user and installed locally (in their home directory).
The idea is that if a user needs some kind of software and is able to build it autonomously then they can set $PATH and other environment variables to make it available to the session. Please note that in ~/.local there are bin, man, lib and other directories, so you can pretty much build, install and run what you need without bothering your sysadmin :)
These days… some package managers might put stuff in there so that they can work without needing root permission and without messing up with the general system.
AFAIK ~/.local/bin is not in any standard. I create this directory manually after installation and add it to my PATH because it makes sense to me, but it is not mandated by the FHS/FreeDesktop/XDG.
None of the mainstream distributions create nor manage that directory for you.
> AFAIK ~/.local/bin is not in any standard. I create this directory manually and add it to my PATH because it makes sense to me, but it is not mandated by the FHS/FreeDesktop/XDG.
> User-specific executable files may be stored in $HOME/.local/bin. Distributions should ensure this directory shows up in the UNIX $PATH environment variable, at an appropriate place.
Most of the $XDG_FOO_DIR dirs are just-in-time created with a `mkdir -p` when something wants to put stuff under them. Given that you're running a program that manipulates the XDG dirs, that program must already exist somewhere else on your system — so it probably doesn't need to write any binaries. It's already installed!
Also, language package ecosystem tooling — the one thing most people touch that could end up installing user-local binaries — is generally just lame and doesn't obey XDG. For just one example, "per-user global" installation of Ruby gems (`gem install --user-install foo`), should properly install the gems under ~/.local/libexec/gem, and then symlink the binstubs of said gem into ~/.local/bin. But it doesn't; it just dumps everything into its own ~/.gem dir and then expects you-the-user to figure out what dir under there to add to your $PATH. Same applies to Rust, Go, Python, and every other language I'm aware of.
One thing you can do that will almost-assuredly use ~/.local/bin/, though, is to intentionally point an autotools configure script at ~/.local as its --prefix. `make install` will then dump binaries into ~/.local/bin.
I'd love to see more language tooling install to ~/.local/bin. I think quite a bit of it predates the widespread use of that XDG directory, so adopting it now would require a migration. I think it'd be worth it, but I also think that's why it hasn't happened yet.
Interesting. I guess 1974 was before "distributions" like BSD even existed, when the OEM bring-up tape was the only OS tape.
But why does Kernighan, on that same page, mention /usr/bin already being a thing? If /usr is what we would today call /home, why does it have a bin subdir?
That only moves the question back a step. Why have a bin user that owns all the userland binaries and keeps them directly under its own home directory? What does that accomplish? Something to do with per-user quotas? Or making `ln /usr/bob/a.out /usr/bin/newcmd` cheap?
I note that there's also a bin group. Was /usr/bin mode 1775, and anyone who should be allowed to install binaries (and update/remove their own installed binaries) was made a member of the bin group?
Also, I note that there's no sbin user. (There is, however, a sys user. But only on Linux, not on BSD...)
sbin is a SYSVR4'ism ... it's complicated. /etc had a bunch of binaries in it like fsck. MACH systems had that as well (well technically it was linked to /private/etc which then linked the binaries to /usr/etc - here's an openstep screenshot I just took: http://9ol.es/usr-etc.png). It's /etc/mkfs /etc/route ... they got moved to /usr/sbin/ and /sbin in R4 - by then, SYSV systems were using /home and the initial purity of every owner of every process being an equivalent class user had been broken by pragmatism.
Unix 1980s history is a complicated mess ... that's why there were multiple conflicting standards groups that tried to fix it. The X/Open group had the "single unix specification" which is different from POSIX and say, the Open software foundation (which did OSF/1). Here's a nice crash course: https://en.wikipedia.org/wiki/Single_UNIX_Specification
Explaining the history with the motivations is probably around 10,000 words. AFAIK it hasn't been written. I'd be one of the handful of wackos who'd actually do that and I haven't seen it.
You misunderstand. My question was why they bothered to make a real "bin" user to have /usr/bin as its homedir; when, if it had just been purely an issue of shoving some binaries over to the /usr drive, they could have just created a /usr/bin owned by root:root and leave it there; or, if they really cared, created a /usr/root for "a secondary root-user homedir that lives on the /usr drive" and then made a /usr/root/bin.
That I don’t know. My guess is that it came later, so that no new user could be called “bin” and thus inherit ownership of those files. There are some historical sources that might shed light on the matter, but someone would have to do some work.
> Utilities used for system administration (and other root-only commands) are stored in /sbin, /usr/sbin, and /usr/local/sbin. /sbin contains binaries essential for booting, restoring, recovering, and/or repairing the system in addition to the binaries in /bin. ...
> Originally, /sbin binaries were kept in /etc. ...
> Deciding what things go into "sbin" directories is simple: if a normal (not a system administrator) user will ever run it directly, then it must be placed in one of the "bin" directories. Ordinary users should not have to place any of the sbin directories in their path.
I don't think it was a security thing, more a usability thing? Why see "reboot" on your path when you can't use it?
Because /bin, /sbin, /usr/local/bin, and /usr/local/sbin all also exist?
(And bevcause GNU autotools uses the variable $PREFIX to abstract away the differences between these — you can `./configure --prefix=/usr` or `./configure --prefix=$HOME/.local` if you like.)
"bin vs sbin" doesn't read right as English — it's almost-but-not-quite like the words are proper nouns that should be capitalized, but instead, it's that they're missing a symbol to indicate that they're paths. The vagueness would become problematic if I wanted to also use "bin dir" in the same paragraph to refer to binary-containing directories generally (i.e. both bin and sbin.)
And the fact that "bin dir" is used that way, to group together both bins and sbins, is is why I didn't say "bin dirs vs sbin dirs." (Note how the prior sentence reads really oddly?)
Putting them each in quotation marks — "bin" vs "sbin" — requires some annoying footwork if you expect you'll need to put larger sections of the paragraph in quotation marks. Most editorial guidelines discourage writers from using quotation marks to mention jargon terms for this reason — especially if the terms will be coming up more than once in a piece.
"/bin vs /sbin" sounds like I'm talking specifically about the ones under /, rather than all of them.
"./bin vs ./sbin" just sounds like I got confused and am talking about something that isn't actually the subject of the parent post.
".../bin vs .../sbin" is pretty good — but "..." here is just doing the same thing "$PREFIX" does but with less guidance as to how it should be interpreted — less ability to Google what it means.
Trust me, I considered every option before settling on writing something as silly as "$PREFIX/bin". "$PREFIX/bin" is weird, but clear. And without an elegant best option, the "weird but clear" option is generally the one you should choose in technical writing. People can't generally call you 20 years later to help parse through a sentence that conflates three different usages of a term, but they can just Google any weird (but well-documented) jargon you've used.
I think "bin vs sbin" reads fine as English given the context of the article, and if you wanted to be more exact you could just use single quotes. I think "bin dir" ad "sbin dir" work fine as well because of the context of the article.
That's just me though. I appreciate the thought you put into your decision as well as your detailed explanation, thank you. I wasn't expecting a detail explanation, but I am always interested in people's reasoning when it's different from mine.
> I understand the historical reasoning for the / vs /usr vs /usr/local split —
That was retconned, actually. My understanding is that the original "/usr" was a second RK05 disk pack added to an early system at Bell Labs for storing stuff owned/created/managed by "users" instead of the system folks. Space was precious in those days and RAID was still a decade in the future. You got your 2MB and had to decide where to mount it and what to put there.
Early Unix filesystems were an ad-hoc wreck, basically. Sun cleaned this up a lot as they moved (or tried to move) to NFS as the primary filesystem for workgroup installations, which is where we get the modern ideas from that you are tagging "historical". I believe (but am not sure) that "sbin" hails from this era.
Then Red Hat came by another decade later and further symmetrized the rules, but in the process essentially obsoleted the distinction between "/" and "/usr", etc...
Isn't the separation of programs into sbin to prevent users without system admin permissions having even read only access to certain programs? Is that not relevant now?
Uhhhh, more than rpm exists. I fail to see a problem that is solved by merging them. I guess some fresh project manager with nothing to do believes they found some low-hanging fruit to get their name on something. ¯\_(ツ)_/¯ It's a waste of time to reinvent this wheel. It will probably make things worse. Many of these ideas come from folks who started with something other than a *nix and are trying to make things feel more like home rather than understand why things are the way they are.
Outside of what the article itself says, even while I was managing large multi-user Linux installs that wasn't really a concern we had. At this point the benefits and concerns are minimal.
I’ve never seen a distro that didn’t grant read permission to everyone for /sbin and /usr/sbin.
In any event, it doesn’t really make sense to me to deny read access for any executables, so long as you don’t also deny read access to Python or a C compiler.
Really, if permission mode bits are the only thing preventing damage from such tools your vulnerability is not within your path or mode bits.
Not really, that can still be done without permissions. You WOULD be able to stop them from knowing the files are there if you remove read permissions to others, but I haven't seen this done.
The main advantage I see of having them separate is that you can remove sbin from user's $PATH to avoid confusion.
If you don't need root to run it, don't install it in */sbin/.
Packages are installed into /usr/bin and /usr/sbin.
Locally-built software (e.g. `./configure --prefix=/; make; make install`) is installed into /usr/local/bin and /usr/local/sbin, which should still always require elevated privileges to write to.
~/.local/bin is convenient but dangerous, because a process should not be able to overwrite itself on disk (unless the user has allowed it to auto-update itself, instead of using a package manager with out-of-band public key retrieval)
This is all probably in the discussions,
A. "Audit all binaries run as root" (TODO lookup the probably CIS control URL for this; look for disk-resident programs that run with elevated privileges)
(When you remove the e(x)ecute bit from a directory, that user/group/other can't list the files in the directory; which is logged to stderr when e.g. `find`
Nowadays we can just do the SBOM tool on everything in every */?bin direcotry and just check those
If "merge sbin into bin" isn't already changed in the FHS spec, I'm -0 to -1 on the idea and don't see really any benefits.
Disadvantage: Users would inadvertantly exec() programs they don't need to run log exec()s of, if /sbin and /bin are merged; there will be more binary enumeration log messages to parse.
Disadvantage: Other distros won't merge /sbin and /bin, so the paths in log messages in support issues and mailing lists would be less universally searchable
78 comments
[ 3.6 ms ] story [ 162 ms ] threadHard disks used to fail pretty often, filesystems had (and still have) bugs. It makes sense to stick your administrative utilities needed to run/recover a given system in its own little less frequently modified filesystem vs. user applications that tend to see more churn/use in general.
It's not strictly a disk space issue.
Good times, I miss working on rhel systems.
An application can be thought of as a combination of the binary, libraries, resources, and state. By default the applications needs to be able to read any of these and be able to read and write the state. Unix's FSH's biggest problem is that it tries to group files together by type instead of grouping files together by application. Nix can technically group things by application, but it is not really standardized so things can get messy and it is still normal for applications to read and write files from anywhere.
Those are features, not bugs... There are reasons for all of those existing and for why you wouldn't want everything in a centralized place.
> NixOS's handling of state is poor. Even ignoring the problem of it all being spread out it is not uncommon for secrets to make it into /nix/store which is world readable.
The secrets handling is a well-known issue and easy to avoid when you're familiar with Nix. Although yes, it can be a problem when people are not aware of it.
And I disagree with NixOS's handling of state being poor. Apart from this "secrets" problem you mentioned, it's not worse than other Linux-based operating systems and in fact it can be argued that it's significantly better.
Just the fact that /nix/store is mounted read-only is, on its own, already a huge improvement over letting users and applications installing and modifying the system in an ad-hoc fashion (good luck when you upgrade your system!).
Not to mention all the other advantages of /nix/store (such as user-installed applications, no package or library conflicts even if you install multiple versions of them, etc).
And as another example, the `stateVersion` feature is something that also makes upgrades a lot more reliable, and I know of no other operating system that has a similar feature.
Atomic configuration changes, booting into specific configurations, and system-wide configuration roll-backs (configurations which also include changes in package versions or even entire OS upgrades), also makes upgrades and configuration changes easy to try / fix if anything goes wrong. No other widely-used operating system has such a reliable configuration change mechanism, as far as I know.
But, you know, if you have other ideas of how things can be improved (apart from your idea of merging user and system-wide state into the same place, which I think makes no sense), I would definitely appreciate to hear it!
It will need to happen sooner or later once distros realize that security is important.
>it's not worse than other Linux-based operating systems
I was not talking in relative terms. NixOS made steps in the right direction, but there is still much more that can be done.
Who knows, maybe something can be fleshed out and improvements come out of it...
That's not really scattered. /nix/store has the generated files. They get linked to /etc if the app requires that kind of thing - often it's just added to the arguments instead. ~/.config is for user preferences (possible to flip at app runtime or static) which are not a system setting. /var is for system services states. Those are very separate use cases and get managed in different ways. The apps should also never be aware that /nix/store is a thing.
To me they all look the same. For example you could have a /nix/state that had a directory for each program. Each program could have its own global and user specific state it could manage from within its directory.
Then you get firejail for more custom apps. You still need that layer and needs to be configured per-app. Would moving config+state into one path really improve the experience? In what way would it be simpler?
NixOS already has that. It's called `/var/lib`.
> Each program could have its own global and user specific state it could manage from within its directory.
The user-specific state is (and should be) in each user's home directory, not scattered around in a bunch of system-wide directories, one for each application...
That's not just for convenience, but also for security (and privacy) reasons.
I'd be interested to know who gets to manage the user-specific state in your proposal. Does each app have it's own per-user directory for each user? i.e. if you have 1000 apps and 10 users, do you need exactly 10,000 user-specific directories? And if not, who gets to create these directories and when are they created/removed?
To be honest, it sounds like you're not aware of all the complexities and implications / consequences of what you're proposing, i.e. it sounds like you didn't think this through. But on the off-chance I'm wrong, I'd be happy to be proven so.
I disagree with this because state is owned by the program so it would make the most sense for it to live in the directory for the program. A user can tell the program that they want to modify the state, but ultimately the program has total ownership of its state. The program should not be leaching into user's home directory to read and write files and should be restricted to directories it owns.
>I'd be interested to know who gets to manage the user-specific state in your proposal.
One way to do it would be for the program to manage it.
>Does each app have it's own per-user directory for each user?
It could choose.
>if you have 1000 apps and 10 users, do you need exactly 10,000 user-specific directories?
This is unrealistic. On the want majority of computers an app will only be run by 1 user. Focusing on handling multiple users is a more special case and is a distraction from the point I'm trying to make.
Really? So a program can access (and modify!) the state of all of its users?
Do you realize how easy it would be to trick my Firefox instance into reading (and even modifying) the state of another user, and even running other code on behalf of another user, given that it would have privileges to do so?
If I understand correctly your proposal, you basically just converted almost every single application bug into a privilege escalation bug.
Not to mention that how do you even define what the same "app" is?
Is Firefox 28 the same app as Firefox 45? And if so, can a user install his own Firefox (like you currently can on NixOS and even other OSes)?
Also, how would you backup all of your own per-user state, given that every app can decide to manage it differently (i.e. storing it in different files and/or directories inside their own per-app directory, or perhaps even in a single per-app database which you wouldn't have access to).
Look, I'm not trying to grill you, I just think you haven't thought this through (but again, I'm happy to be proven wrong).
> This is unrealistic. On the want majority of computers an app will only be run by 1 user. Focusing on handling multiple users is a more special case and is a distraction from the point I'm trying to make.
What does this even mean? We're talking about user-specific state. So there's different state for each user, which means there are multiple users, by definition!
So how is focusing on multiple users a distraction? It's the whole point...
As I mentioned, on NixOS, the system-wide mutable state is already stored on /var/lib/<app>, which is what you are (redundantly) proposing. So we're just arguing about how to handle user-specific state, nothing else.
I never said that. The same binary can be run by multiple users and the user running it affects what files it can read and write.
>Not to mention that how do you even define what the same "app" is?
The user has an idea on what the same app is. Unfortunately Nix does not properly understand this concept.
>Also, how would you backup all of your own per-user state, given that every app can decide to manage it differently
You could back up the entire state visible to the user.
>What does this even mean?
Having a single set of state for each program would be good enough for most users.
>As I mentioned, on NixOS, the system-wide mutable state is already stored on /var/lib/<app>, which is what you are (redundantly) proposing.
That is one place. Some programs don't use it. It's not a centralized place of state for every program.
What you're proposing is not even possible with standard Unix permissions (without leaving a huge security hole). You're suggesting that each program can create the state for each user inside the program's directory but somehow cannot read or modify the state for other users.
This suggests the program runs with each user's privileges and therefore these directories would require the sticky bit for state creation (like /tmp does) but this would not prevent a user from roguely creating the state for another user (this is why files in /tmp should be created with random names, they should not be predictable).
Or maybe you're suggesting that all apps should run with setuid privileges (so that they can create the state for each user, but users themselves couldn't) but we already know that would be absolutely terrible for security.
So something like AppArmor (which AFAIK has security flaws) or even SELinux would be required to implement this scheme, I believe, if it's possible at all.
SELinux, AppArmor and other jail-like mechanisms can be a huge burden to manage, especially since most apps would require a substantial number of exceptions to a default security policy (think GUI apps, or apps that require access to a user's files, for example)...
Since most apps do require read/write access to a user's files, or permission to show things on a screen, I'm not sure if you're gaining much security with your scheme. There would be almost infinite ways for an app to exfiltrate or modify a user's files or trick the user into doing things on behalf of the app, I think (like running other apps...).
And if you're implementing a restrictive security policy with something like SELinux or even some kind of container/jail, you might as well do the same but without having to change each and every app to use the weird per-app directory containing the state of all users. This would save you a huge amount of work and potential security vulnerabilities, since private home directories are already secure by default (in terms of protecting a user from other users) and restricting an app not to access the state of other apps could be equally done when using home directories.
> The user has an idea on what the same app is. Unfortunately Nix does not properly understand this concept.
So a user can decide what "Firefox" is? Then what prevents a user from installing a rogue Firefox that exfiltrates the other users' state, in your scheme?
> That is one place. Some programs don't use it. It's not a centralized place of state for every program.
It is for programs that run system-wide and they should be using it.
Which system-wide programs are not using /var/lib for their mutable state?
It's not like they have many other places to save their state... I mean, there is /var/cache but that's for state that can be removed without data loss.
That sounds horrible. NixOS sounded kind of interesting, but if that's true it kills any interest I had in it.
On the other hand, your personal files I your home directory are left alone; they don't get put in the nix store. An environment that needs access to your personal files (such as your desktop) simply has your home directory mounted into it.
That's what a file system is.
Source: https://nixos.wiki/wiki/Overview_of_the_NixOS_Linux_distribu...
https://wiki.gobolinux.org/Overview/GoboLinux-Filesystem-Hie...
While I was looking up that link, I found the original discussion for Fedora 10 (2008) about adding /usr/sbin to $PATH which was a little more controversial (at the time): https://lists.fedoraproject.org/archives/list/devel@lists.fe...
I understand the historical reasoning for the / vs /usr vs /usr/local split —
• / was a mount of the machine vendor's installation-engineering-provided bring-up tape — a tape that could usually be safely unmounted once boot was complete and the kernel + persistent daemons had loaded into RAM. (We still have something akin to this in Linux land in the form of the initramfs!)
• /usr was a read-only-mounted OS distribution, either spooled onto the same tape drive that had been reading the rootfs tape a moment before, or (later) over NFS from a high-read-IOPS shared disk server, where in either case the mountpoint was expected to never go away while the OS was running, and therefore the loader could spawn binaries from here using flushable memory-mapped disk IO (allowing far more running processes to fit within the machine's limited memory — important for time-sharing!);
• /usr/local was a separate site-specific mount of stuff added by the local sysadmin — often created as a separate "tacked on the end" partition on the OS distribution tape (you could just attach more tape on the end of the reel, and then format it!) Later, once machine-local hard disks were a thing but Unix systems were still generally mounting /usr over NFS, /usr/local evolved to be the read-only partition of the machine-local HD (and such systems gained a matching read-write local-disk partition, /var.)
But $PREFIX/bin and $PREFIX/sbin were, as far as I know, always just regular directories under the same $PREFIX-level mountpoint, never separate mountpoints of their own.
The only difference I can even think of between a $PREFIX/bin and a $PREFIX/sbin dir, is that the $PREFIX/sbin dir wasn't in $PATH for most users. But surely that was never a real line of defense for security?
The idea is that if a user needs some kind of software and is able to build it autonomously then they can set $PATH and other environment variables to make it available to the session. Please note that in ~/.local there are bin, man, lib and other directories, so you can pretty much build, install and run what you need without bothering your sysadmin :)
These days… some package managers might put stuff in there so that they can work without needing root permission and without messing up with the general system.
None of the mainstream distributions create nor manage that directory for you.
Quoting https://specifications.freedesktop.org/basedir-spec/basedir-... :
> User-specific executable files may be stored in $HOME/.local/bin. Distributions should ensure this directory shows up in the UNIX $PATH environment variable, at an appropriate place.
Still, I haven't seen any distro create or put it in PATH in the 24 years I've been running Linux.
Also, language package ecosystem tooling — the one thing most people touch that could end up installing user-local binaries — is generally just lame and doesn't obey XDG. For just one example, "per-user global" installation of Ruby gems (`gem install --user-install foo`), should properly install the gems under ~/.local/libexec/gem, and then symlink the binstubs of said gem into ~/.local/bin. But it doesn't; it just dumps everything into its own ~/.gem dir and then expects you-the-user to figure out what dir under there to add to your $PATH. Same applies to Rust, Go, Python, and every other language I'm aware of.
One thing you can do that will almost-assuredly use ~/.local/bin/, though, is to intentionally point an autotools configure script at ~/.local as its --prefix. `make install` will then dump binaries into ~/.local/bin.
Even in 1994 this was still common, https://archive.org/details/learningunix0000gard/page/48/mod...
But why does Kernighan, on that same page, mention /usr/bin already being a thing? If /usr is what we would today call /home, why does it have a bin subdir?
Look at row #3.
I note that there's also a bin group. Was /usr/bin mode 1775, and anyone who should be allowed to install binaries (and update/remove their own installed binaries) was made a member of the bin group?
Also, I note that there's no sbin user. (There is, however, a sys user. But only on Linux, not on BSD...)
The invention of /sbin/ /usr/sbin is all R4, see https://archive.org/details/unixsystemvrelea0000unse_t8i3/pa...
Unix 1980s history is a complicated mess ... that's why there were multiple conflicting standards groups that tried to fix it. The X/Open group had the "single unix specification" which is different from POSIX and say, the Open software foundation (which did OSF/1). Here's a nice crash course: https://en.wikipedia.org/wiki/Single_UNIX_Specification
Explaining the history with the motivations is probably around 10,000 words. AFAIK it hasn't been written. I'd be one of the handful of wackos who'd actually do that and I haven't seen it.
> Utilities used for system administration (and other root-only commands) are stored in /sbin, /usr/sbin, and /usr/local/sbin. /sbin contains binaries essential for booting, restoring, recovering, and/or repairing the system in addition to the binaries in /bin. ...
> Originally, /sbin binaries were kept in /etc. ...
> Deciding what things go into "sbin" directories is simple: if a normal (not a system administrator) user will ever run it directly, then it must be placed in one of the "bin" directories. Ordinary users should not have to place any of the sbin directories in their path.
I don't think it was a security thing, more a usability thing? Why see "reboot" on your path when you can't use it?
Why did you go to so much effort to avoid typing 'usr'?
(And bevcause GNU autotools uses the variable $PREFIX to abstract away the differences between these — you can `./configure --prefix=/usr` or `./configure --prefix=$HOME/.local` if you like.)
And the fact that "bin dir" is used that way, to group together both bins and sbins, is is why I didn't say "bin dirs vs sbin dirs." (Note how the prior sentence reads really oddly?)
Putting them each in quotation marks — "bin" vs "sbin" — requires some annoying footwork if you expect you'll need to put larger sections of the paragraph in quotation marks. Most editorial guidelines discourage writers from using quotation marks to mention jargon terms for this reason — especially if the terms will be coming up more than once in a piece.
"/bin vs /sbin" sounds like I'm talking specifically about the ones under /, rather than all of them.
"./bin vs ./sbin" just sounds like I got confused and am talking about something that isn't actually the subject of the parent post.
".../bin vs .../sbin" is pretty good — but "..." here is just doing the same thing "$PREFIX" does but with less guidance as to how it should be interpreted — less ability to Google what it means.
Trust me, I considered every option before settling on writing something as silly as "$PREFIX/bin". "$PREFIX/bin" is weird, but clear. And without an elegant best option, the "weird but clear" option is generally the one you should choose in technical writing. People can't generally call you 20 years later to help parse through a sentence that conflates three different usages of a term, but they can just Google any weird (but well-documented) jargon you've used.
That's just me though. I appreciate the thought you put into your decision as well as your detailed explanation, thank you. I wasn't expecting a detail explanation, but I am always interested in people's reasoning when it's different from mine.
That was retconned, actually. My understanding is that the original "/usr" was a second RK05 disk pack added to an early system at Bell Labs for storing stuff owned/created/managed by "users" instead of the system folks. Space was precious in those days and RAID was still a decade in the future. You got your 2MB and had to decide where to mount it and what to put there.
Early Unix filesystems were an ad-hoc wreck, basically. Sun cleaned this up a lot as they moved (or tried to move) to NFS as the primary filesystem for workgroup installations, which is where we get the modern ideas from that you are tagging "historical". I believe (but am not sure) that "sbin" hails from this era.
Then Red Hat came by another decade later and further symmetrized the rules, but in the process essentially obsoleted the distinction between "/" and "/usr", etc...
In any event, it doesn’t really make sense to me to deny read access for any executables, so long as you don’t also deny read access to Python or a C compiler.
Really, if permission mode bits are the only thing preventing damage from such tools your vulnerability is not within your path or mode bits.
The main advantage I see of having them separate is that you can remove sbin from user's $PATH to avoid confusion.
Packages are installed into /usr/bin and /usr/sbin.
Locally-built software (e.g. `./configure --prefix=/; make; make install`) is installed into /usr/local/bin and /usr/local/sbin, which should still always require elevated privileges to write to.
~/.local/bin is convenient but dangerous, because a process should not be able to overwrite itself on disk (unless the user has allowed it to auto-update itself, instead of using a package manager with out-of-band public key retrieval)
This is all probably in the discussions,
A. "Audit all binaries run as root" (TODO lookup the probably CIS control URL for this; look for disk-resident programs that run with elevated privileges)
A.1. Find setuid and setgid files:
A.2. Find setcap binariesA.3. Find extended filesystem attribute permissions with e.g. getfacl
A.4. List everything in */sbin that only root users should need to run (*)
A.5. Find everything that's `chown root; chmod o-x`;
(When you remove the e(x)ecute bit from a directory, that user/group/other can't list the files in the directory; which is logged to stderr when e.g. `find`
Nowadays we can just do the SBOM tool on everything in every */?bin direcotry and just check those
If "merge sbin into bin" isn't already changed in the FHS spec, I'm -0 to -1 on the idea and don't see really any benefits.
Filesystem Hierarchy Standard: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
Disadvantage: Users would inadvertantly exec() programs they don't need to run log exec()s of, if /sbin and /bin are merged; there will be more binary enumeration log messages to parse.
Disadvantage: Other distros won't merge /sbin and /bin, so the paths in log messages in support issues and mailing lists would be less universally searchable