I've only used modern immutable Linux (Alpine, MicroOS) and wondered why of all places `/var/` was chosen as the location for rw stuff. It's fun to be reminded that there was of course a time when an immutable OS was the default, and you ran it off of floppies. So there's a lot of history to using `/var/` for that. Guess we've come full circle!
macOS has all of that (mostly inherited from NeXTSTEP which was significantly based on 4.3/4.4BSD). It's hidden by default in the GUI, visible in Terminal.
Nowadays most end users just use /usr/local or /opt/local or whatever is managed by Homebrew or Macports.
This is what happens when a system is designed by multiple people and companies over a long period of time. An amalgam of ideas which are there just because. There's no reason Linux should be like this. e.g., see https://gobolinux.org/ which has more sane dirs.
1. The title says “understanding sbin” but the content gives zero understanding of that. If someone has a historical explanation, please provide it.
2. “Then somebody decided
/usr/local wasn't a good place to install new packages, so let's add /opt”
Not exactly. /usr/local exists so you don’t accidentally mess up your distro/package manager by changing its files. It’s “local” to your installation. But it is still structured — /usr/local/bin, /usr/local/lib, etcetera — divided into binaries, shared libraries, manpages.
Whereas /opt has no structure. It’s “the wild west”…application binaries, libraries, configuration, data files, etcetera with no distinction. Apps with “universal” packaging, or sometimes secondary package managers.
For example /usr/local/bin is normally part of PATH, but /opt is not (unless eg homebrew adds it to your bashrc).
On a similar note: just the other day I was thinking about how the Unixy systems I used 20+ years ago used to nudge/push you toward creating several actual partitions during installation. Maybe /, /usr, swap… maybe one or two more? IIRC, I think some of the BSDs, at least, maybe still do? Always seemed weird and suboptimal to me for most installations, but I remember being told by graybeards at the time that it was the Right Way.
I have always made /home a separate partition.
This makes it so much easier to reinstall and/or wipe out a distro and install a new one. All of my files are left undisturbed.
Also /bin vs /sbin believe is that the latter is meant for statically linked binaries such that if your system is corrupted these at least will keep working.
Practically in this century if I was starting a new OS I would set it up like so:
/bin for all system binaries. Any binary from a package installed by the OS package manager lived here.
/lib same but for shared libraries
/var for variable data. This is where you would put things like your Postgres data files.
/tmp for temporary files.
/home as usual.
/dev as usual.
/boot as usual
/etc as usual
/usr would be what /usr/local is on most systems. So /usr/bin is binaries not installed by the OS package manager. /usr/etc is where you put config files for packages not installed by the package manager and so on.
Get rid of /usr/local and /sbin.
/media replaces /mnt entirely (or vice versa).
Ditch /opt and /srv
Add /sub for subsystems: container overlays should live here. This would allow the root user (or a docker group, etc.) to view the container file system, chroot into it, or run a container on it.
Then again, nobody gave me a PDP-11 to decide so my vote doesn’t count :)
> Also /bin vs /sbin believe is that the latter is meant for statically linked binaries such that if your system is corrupted these at least will keep working.
I think that became the rationale for /[s]bin vs. /usr/[s]bin (although based on the linked article, that may have been retconned a bit).
You were supposed to keep your root/boot filesystem very small and mostly read-only outside major updates. That meant that you could boot to a small amount of utilities (e.g. fsck) that would let you repair /usr or any other volume if it became corrupted.
I think the other poster is correct that stuff like fsck is supposed to go into /sbin because it is a "system" binary (but also statically linked since /usr/lib isn't mounted yet) and doesn't make sense to have in user $PATHs since nobody other than root should really be running that one.
Regardless, this is all deeply meaningless these days, particularly if you are running "ephemeral" infrastructure where if anything goes that wrong you just repave it all and start over.
If I was starting a new system layout, I wouldn't have every package smush its files together with everyone else's into a single shared directory hierarchy. /opt would reign supreme, and we already have pkg-config to deal with that sort of layout.
Good point. It isn’t variable data as in it only changes on package updates. But it isn’t user data either, so it doesn’t belong in /usr. I guess that one should have been in /lib/share and /lib/man. Or it could be a brand new directory called /doc which contains the documentation type content.
/usr/lib contents belongs in /lib if the contents is installed by the package manager. That way /usr/lib is what the user installed.
This post gets some of the details wrong. /usr/local is for site-local software - e.g. things you compile yourself, i.e in the case of the BSDs the ports collection - things outside the base system. (They may be compiled for you).
Since Linux has no concept of a base system, it's a stand-alone kernel with a hodgepodge of crap around it - this distinction makes no sense on Linux.
/opt is generally for software distros for which you don't have source; only binaries. Like commercial software packages. More common on Real UNIX(R) because most Linux users outside enterprise aren't running commercial software. You're putting your $500k EDA software under /opt.
Now I get what the folks using FreeBSD typically like to point to as a reason why they prefer FreeBSD over Linux because there is a clear distinction between the base system and userland.
> Linux has no concept of a base system, it's a stand-alone kernel with a hodgepodge of crap around it
Good grief. How does this end up as the top comment on HN of all places? I'll bet anything that this author also thinks that systemd is way too opinionated and unified and that the system needs a less coupled set of init code.
Edit to be at least a tiny bit more productive: the Linux Filesystem Hierarchy Standard is about to pop the cork on its thirty second birthday. It's likely older than most of the people upvoting the post I responded to. https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
To wit: that's outrageous nonsense, and anyone who know anything about how a Linux distro is put together (which I thought would have included most of the readers here, but alas) would know that.
1. The history of /usr subdirectories is a lot more complex than that. There was a /usr/lbin once, for example.
1. /usr/local is not where third party softwares from packages/ports go on "the BSDs". On NetBSD, they go in /usr/pkg instead, again exemplifying that this is quite complex through history and across operating systems.
Here [1] is a related trick in the old Unix to run either `foo`, `/bin/foo` or `/usr/bin/foo` (apparently before `PATH` convention existed):
char string[10000];
strp = string;
for (i=0; i<9; i++)
*strp++ = "/usr/bin/"[i];
p = *argv++;
while(*strp++ = *p++);
// string == "/usr/bin/foo"
execv(string+9, args); // foo (execv returns only in case of error, i.e. when foo does not exist)
execv(string+4, args); // /bin/foo
execv(string, args); // /usr/bin/foo
Nowadays I think packages should turn to portable applications isolated within their own directories. Those directories would have an standard libraries directory that the application would use.
Latter, if desired, the system, could override those libraries with another ones (newer compatible or patched), more thinking is needed about this. The key, from the process point of view, would to limit the access of such process to their own directories and some very limited system only local services by default,
And to extend this permissions, each binary in such directory would need to be in companion of a permissions request file that would require the approbation from the user or the system defaults patterns (each distro would have a point of view I guess), in the aim of improve process isolation and system, drivers, services access permissions.
This would need also restructure the console philosophy, how can manage the processes, and so on, that would need a big restructuration.
I mean, anyway people is duplicating space with containers trying to isolate process, remark in trying.
I know this is unrealistic due the deep change it would suppose, so consider I'm just thinking out loud.
PS: If you answer it already exists with AppArmor, SELinux, etc, then you did not understood the root of the issue with such modules.
Sometime around 2000 someone decided that /bin and /sbin isn't enough to boot and mount the rest of the system, so they added further complexity: an initrd/initramfs that does the basic job of /bin and /sbin. They had to complicate the kernel build process, the kernel update, the bootloader, the kernel command line and for what? Just because they didn't want the kernel to have the storage drivers built-in?
So the /bin /sbin became redundant.
Sometime around 2020 someone observed that no current Linux can boot without /usr anyway. So what did they do? Move everything from /usr to / and drop the whole /usr legacy? Noooo, that would be too simple. Move / to /usr. And because that is still too simple, also move /bin, /sbin and /usr/sbin to /usr/bin, and then keep symlinks at the old locations because who's gonna fix hardcoded paths in 99% of all Linux apps anyway??
Oh, how I wish I was born in the '60s, when the world was still sane.
/ has to be writeable (or have separate writeable mounts under it), /usr doesn't. The reasons for unifying under /usr are clearly documented and make sense and it's incredibly tedious seeing people complain about it without putting any effort into understanding it.
busybox in Alpine Linux has for example `ps` builtin. If you install ps with `apk add ps` to get the full version, it will remove the symlink for /bin/ps and replace it with the one you installed.
You need to read up on the purpose of busybox. It is not something that the kernel people has decided upon. It is an initiative of an group of people who needed some tools onto a single floppy.
/bin/ps on a Debian distro is 154522 bytes.
The whole busybox in Alpine Linux is 804616 bytes and contains a whole lot more than just ps.
The next logical evolution is to get rid of directories and put everything in /. This will simplify a lot of the build process. /usr/include and /usr/lib are already a mess (on linux).
If you had been born in the 1960s, you might well have learned by dint of being alive at the time that the world underneath /usr was pretty complicated in the 1970s, 1980s, and 1990s; that /etc was where some of the things that were used to boot the system once went; and that the tale of sbin is complex and slightly sad.
The tale that things were simple until they went to pot in 2000 is wholly ahistoric.
Rob Landley is right. Others realised this too - see GoboLinux. But some regular distributions too.
Slapping it down as "FHS is now a standard" does not change anything. People will ask why it is suddenly a standard when it hasn't made any sense at all whatsover. bin versus sbin is also pointless. Inertia is one primary reason why nobody fixes things usually.
This is low-effort fantasy history. It may be directionally correct, but why bother when you don't care about the details? From analyzing the UNIX manuals and other old files we get the following (not fully complete) picture:
We'll skip PDP-7 UNIX, no hierarchical file system yet.
UNIX v1 on the PDP-11 had an RF11 fixed head disk (1mb) for / and swap, and an RK05 moving head disk (2.5mb) for /usr (the user directories)
By v2 they had added a second RK05 at /sys for things like the kernel, manual pages, and system language stuff like the c compiler and m6.
By v3 they added yet another RK05 at /crp for, well, all sorts of crap (literally), including yacc apparently. /usr/bin is mentioned here for the first time.
I don't feel like looking up when sbin was first introduced but it is not a Bell Labs thing. possibly BSD or AT&T UNIX? Binaries that one would normally not want to run were kept in /etc, which includes thing like init, mount, umount, getty, but also the second pass of the assembler (as2), or helpers like glob.
Also i don't know when /home became canonical but at Bell Labs it was never a thing (plan 9 has user directories in /usr where they had always belonged logically).
The lib situation is more difficult. Looks like it started with /usr/lib. By v3 we find the equivalent directory as /lib, where it contains the two passes of the C compiler (no optimization pass back then), C runtime and lib[abc].a (assembler, B, C libraries respectively). /usr/lib had been repurposed for non-object type libraries, think text-preparation and typesetting.
By v4 the system had escaped the labs (see the recent news) and at that point everyone modified the system to their taste anyway. Perhaps it should be noted that the v7 distribution (which is the first that is very clearly the ancestor of every modern UNIX) has no /usr/bin, only /bin. /lib and /usr/lib are split however.
These are just some rough notes and due to a lack of early material they're still not as accurate as i would like. Also UNIX ran on more than one machine even in the early days (the manuals mention the number of installations) so there must have been some variation anyway. Something I'd like to know in particular is when and where RP03 disk drives were used. These are pretty huge in comparison to the cute RK05s.
Does anyone know why, when Lennart and friends wrote their XDG Base Directory Specification, they decided that each user should replicate /usr/local/ subdirectories under $HOME/.local/?
Doesn't being under $HOME make .local redundant? I guess one could argue for binaries going in an architecture-specific subdirectory if $HOME was on a shared filesystem, but that's not what's being done here.
To me, $HOME/.local/share and its siblings are just a needless level of indirection, forcing me to jump through an extra hoop every time I want to access what's in there.
(I know it's sometimes possible to override it with an environment variable, but the predictably spotty support for those overrides means I would then have to look for things in two places. I think sensible defaults would be nicer.)
Plan 9 (a Bell Labs successor to Unix) did away with the whole bin, sbin, usr/sbin thing and its shell only looked in /bin. How things got into /bin is a different story.
For me this was an eye-opener. I kept trying to wrap my head around all these different paths and "standards" because I thought it was correct and deliberately designed. Looking back through the history this doesn't seem to be the case; I feel much better for being confused by all the different PATH conventions and strict hierarchies.
The reason you didn't just drop stuff in /usr/local? Space.
One of our devs was also a gimp contributor, and he dropped gimp into /usr/local and filled up the filesystem. And back then package managers didn't exist, so you had to read the makefile and hope you didn't remove anything that was shared
/opt/gimp or /usr/local/gimp.
Local because in some places they mounted an nfs share, and local was local to you.
Some people think today's file hierarchy is complicated. .That's amusing.
I worked at an R&D center where we had hundreds of UNIX systems orf all types(i.e. Sun, Ultrix, HP, Symbolics, etc.) We also had Sun 2's , 3's and 4's - each with different CPU's/architectures and incompatible binaries. Some Suns had no disks at all. And with hundreds of systems, we literally had a hundred different servers across the entire site.
I would compile a program for a Sun 3, and needed a way to install the program once, for use on hundreds of computers. Also teams of people on dozens of different computers needed to share files with each other.
I did consulting work for a place with Sun 2's and 4's, AIX, HP-UX, Xenix, and SCO. There was NFS, Netware, and UUCP all cheerfully coexisting on an IPv4 network that used a public Class C for NATed internal use. (It's now a zombie IP range that doesn't do anything useful.)
Later, I wrote code a university that had even more heterogeneous clusters.
The problem today is the siloification and bias against system diversity that doesn't account for proper software configuration management and support of multiple platforms. Portability is a dying art.
One complication caused by shared libraries was the security threat. An executable using a shared library allowed the user to execute with a different (updated) library without recompilation.
This is a security threat, especially with SETUID programs. If you could change the library, you could install new code and gain privileged access.
This was why /usr/sbin was created - all of the programs there were compiled with static libraries.
This is also around the same time Vista started enforcing the AppData and ProgramData folder redirect. It was a messy time for all developers. IMHO they made the right decision by enforcing that redirect as we now know where every file should be in a Windows program.
45 comments
[ 2.8 ms ] story [ 76.9 ms ] threadAdditional info: many rules from many places are now in force that maintain the historical structure.
Mac OS?
Nowadays most end users just use /usr/local or /opt/local or whatever is managed by Homebrew or Macports.
Well...
:'-)And I thought we just got over the systemd drama…
Speaking of things which are needlessly complex, I'm reminded of this classic post on the tortured history of the browser User-Agent header:
https://webaim.org/blog/user-agent-string-history/
Highly recommended!
2. “Then somebody decided /usr/local wasn't a good place to install new packages, so let's add /opt”
Not exactly. /usr/local exists so you don’t accidentally mess up your distro/package manager by changing its files. It’s “local” to your installation. But it is still structured — /usr/local/bin, /usr/local/lib, etcetera — divided into binaries, shared libraries, manpages.
Whereas /opt has no structure. It’s “the wild west”…application binaries, libraries, configuration, data files, etcetera with no distinction. Apps with “universal” packaging, or sometimes secondary package managers.
For example /usr/local/bin is normally part of PATH, but /opt is not (unless eg homebrew adds it to your bashrc).
Practically in this century if I was starting a new OS I would set it up like so:
/bin for all system binaries. Any binary from a package installed by the OS package manager lived here.
/lib same but for shared libraries
/var for variable data. This is where you would put things like your Postgres data files.
/tmp for temporary files.
/home as usual.
/dev as usual.
/boot as usual
/etc as usual
/usr would be what /usr/local is on most systems. So /usr/bin is binaries not installed by the OS package manager. /usr/etc is where you put config files for packages not installed by the package manager and so on.
Get rid of /usr/local and /sbin.
/media replaces /mnt entirely (or vice versa).
Ditch /opt and /srv
Add /sub for subsystems: container overlays should live here. This would allow the root user (or a docker group, etc.) to view the container file system, chroot into it, or run a container on it.
Then again, nobody gave me a PDP-11 to decide so my vote doesn’t count :)
I think that became the rationale for /[s]bin vs. /usr/[s]bin (although based on the linked article, that may have been retconned a bit).
You were supposed to keep your root/boot filesystem very small and mostly read-only outside major updates. That meant that you could boot to a small amount of utilities (e.g. fsck) that would let you repair /usr or any other volume if it became corrupted.
I think the other poster is correct that stuff like fsck is supposed to go into /sbin because it is a "system" binary (but also statically linked since /usr/lib isn't mounted yet) and doesn't make sense to have in user $PATHs since nobody other than root should really be running that one.
Regardless, this is all deeply meaningless these days, particularly if you are running "ephemeral" infrastructure where if anything goes that wrong you just repave it all and start over.
/usr/lib contents belongs in /lib if the contents is installed by the package manager. That way /usr/lib is what the user installed.
Since Linux has no concept of a base system, it's a stand-alone kernel with a hodgepodge of crap around it - this distinction makes no sense on Linux.
/opt is generally for software distros for which you don't have source; only binaries. Like commercial software packages. More common on Real UNIX(R) because most Linux users outside enterprise aren't running commercial software. You're putting your $500k EDA software under /opt.
"some" is an understatement.
Good grief. How does this end up as the top comment on HN of all places? I'll bet anything that this author also thinks that systemd is way too opinionated and unified and that the system needs a less coupled set of init code.
Edit to be at least a tiny bit more productive: the Linux Filesystem Hierarchy Standard is about to pop the cork on its thirty second birthday. It's likely older than most of the people upvoting the post I responded to. https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
To wit: that's outrageous nonsense, and anyone who know anything about how a Linux distro is put together (which I thought would have included most of the readers here, but alas) would know that.
It is open-source, and I can get source files, but it’s precompiled…
1. The history of /usr subdirectories is a lot more complex than that. There was a /usr/lbin once, for example.
1. /usr/local is not where third party softwares from packages/ports go on "the BSDs". On NetBSD, they go in /usr/pkg instead, again exemplifying that this is quite complex through history and across operating systems.
[0]: https://www.psychologytoday.com/us/blog/thinking-makes-it-so...
Latter, if desired, the system, could override those libraries with another ones (newer compatible or patched), more thinking is needed about this. The key, from the process point of view, would to limit the access of such process to their own directories and some very limited system only local services by default,
And to extend this permissions, each binary in such directory would need to be in companion of a permissions request file that would require the approbation from the user or the system defaults patterns (each distro would have a point of view I guess), in the aim of improve process isolation and system, drivers, services access permissions.
This would need also restructure the console philosophy, how can manage the processes, and so on, that would need a big restructuration.
I mean, anyway people is duplicating space with containers trying to isolate process, remark in trying.
I know this is unrealistic due the deep change it would suppose, so consider I'm just thinking out loud.
PS: If you answer it already exists with AppArmor, SELinux, etc, then you did not understood the root of the issue with such modules.
So the /bin /sbin became redundant.
Sometime around 2020 someone observed that no current Linux can boot without /usr anyway. So what did they do? Move everything from /usr to / and drop the whole /usr legacy? Noooo, that would be too simple. Move / to /usr. And because that is still too simple, also move /bin, /sbin and /usr/sbin to /usr/bin, and then keep symlinks at the old locations because who's gonna fix hardcoded paths in 99% of all Linux apps anyway??
Oh, how I wish I was born in the '60s, when the world was still sane.
busybox in Alpine Linux has for example `ps` builtin. If you install ps with `apk add ps` to get the full version, it will remove the symlink for /bin/ps and replace it with the one you installed.
You need to read up on the purpose of busybox. It is not something that the kernel people has decided upon. It is an initiative of an group of people who needed some tools onto a single floppy.
/bin/ps on a Debian distro is 154522 bytes. The whole busybox in Alpine Linux is 804616 bytes and contains a whole lot more than just ps.
https://en.wikipedia.org/wiki/BusyBox https://busybox.net/
The tale that things were simple until they went to pot in 2000 is wholly ahistoric.
Slapping it down as "FHS is now a standard" does not change anything. People will ask why it is suddenly a standard when it hasn't made any sense at all whatsover. bin versus sbin is also pointless. Inertia is one primary reason why nobody fixes things usually.
https://joeldare.com/where-i-put-personal-binaries-in-macos
We'll skip PDP-7 UNIX, no hierarchical file system yet.
UNIX v1 on the PDP-11 had an RF11 fixed head disk (1mb) for / and swap, and an RK05 moving head disk (2.5mb) for /usr (the user directories)
By v2 they had added a second RK05 at /sys for things like the kernel, manual pages, and system language stuff like the c compiler and m6.
By v3 they added yet another RK05 at /crp for, well, all sorts of crap (literally), including yacc apparently. /usr/bin is mentioned here for the first time.
I don't feel like looking up when sbin was first introduced but it is not a Bell Labs thing. possibly BSD or AT&T UNIX? Binaries that one would normally not want to run were kept in /etc, which includes thing like init, mount, umount, getty, but also the second pass of the assembler (as2), or helpers like glob. Also i don't know when /home became canonical but at Bell Labs it was never a thing (plan 9 has user directories in /usr where they had always belonged logically).
The lib situation is more difficult. Looks like it started with /usr/lib. By v3 we find the equivalent directory as /lib, where it contains the two passes of the C compiler (no optimization pass back then), C runtime and lib[abc].a (assembler, B, C libraries respectively). /usr/lib had been repurposed for non-object type libraries, think text-preparation and typesetting.
By v4 the system had escaped the labs (see the recent news) and at that point everyone modified the system to their taste anyway. Perhaps it should be noted that the v7 distribution (which is the first that is very clearly the ancestor of every modern UNIX) has no /usr/bin, only /bin. /lib and /usr/lib are split however.
These are just some rough notes and due to a lack of early material they're still not as accurate as i would like. Also UNIX ran on more than one machine even in the early days (the manuals mention the number of installations) so there must have been some variation anyway. Something I'd like to know in particular is when and where RP03 disk drives were used. These are pretty huge in comparison to the cute RK05s.
I am also totally adding a /crp directory to my next system.
Doesn't being under $HOME make .local redundant? I guess one could argue for binaries going in an architecture-specific subdirectory if $HOME was on a shared filesystem, but that's not what's being done here.
To me, $HOME/.local/share and its siblings are just a needless level of indirection, forcing me to jump through an extra hoop every time I want to access what's in there.
(I know it's sometimes possible to override it with an environment variable, but the predictably spotty support for those overrides means I would then have to look for things in two places. I think sensible defaults would be nicer.)
For me this was an eye-opener. I kept trying to wrap my head around all these different paths and "standards" because I thought it was correct and deliberately designed. Looking back through the history this doesn't seem to be the case; I feel much better for being confused by all the different PATH conventions and strict hierarchies.
One of our devs was also a gimp contributor, and he dropped gimp into /usr/local and filled up the filesystem. And back then package managers didn't exist, so you had to read the makefile and hope you didn't remove anything that was shared
/opt/gimp or /usr/local/gimp.
Local because in some places they mounted an nfs share, and local was local to you.
I worked at an R&D center where we had hundreds of UNIX systems orf all types(i.e. Sun, Ultrix, HP, Symbolics, etc.) We also had Sun 2's , 3's and 4's - each with different CPU's/architectures and incompatible binaries. Some Suns had no disks at all. And with hundreds of systems, we literally had a hundred different servers across the entire site.
I would compile a program for a Sun 3, and needed a way to install the program once, for use on hundreds of computers. Also teams of people on dozens of different computers needed to share files with each other.
This was before SSH. We had to use NFS.
It was fairly seamless and .... interesting.
Later, I wrote code a university that had even more heterogeneous clusters.
The problem today is the siloification and bias against system diversity that doesn't account for proper software configuration management and support of multiple platforms. Portability is a dying art.
This is a security threat, especially with SETUID programs. If you could change the library, you could install new code and gain privileged access.
This was why /usr/sbin was created - all of the programs there were compiled with static libraries.