Where I think many people get weaker abstractions of this in their mind is where the OP hints:
> and therefore a more accurate description of this feature is Everything is a file descriptor
The filesystem is where files go to get names, and to be organized. But critically, not everything is a normal file, which I think is where people experience the most "the abstraction is leaky!" feelings. I don't think the abstraction needs everything to be a normal file, but I think there is still a lot of use in having a unified namespace for all the various types of files, normal or otherwise. (As opposed to having disparate namespaces for files of different types.)
For the "file descriptor" half, the value is being able to wait on events/event completion for a set of heterogeneous file descriptors, but boy did it take a while to get there with usable interfaces for that.
It's fine for local IPC like Unix domain sockets. For remote IPC, obviously the machines don't share a common space they can just access as a file for rendezvous, and so we don't use that API.
We have socket, connect|{bind, listen, accept}, read, write.
For connectionless, send, recv.
Sockets are not part of "everything is a file" but of "everything is a file descriptor".
>It's fine for local IPC like Unix domain sockets.
I disagree. I should be able to just call a function. I don't want to bother figuring out the network protocol that the process decided on. It's too low level to expose.
Yes! There are a lot of benefits to the 'everything is a file' philosophy, but files aren't very good APIs for a structured data. The perfect example is the stuff in /proc and /sys. Making it all 'just a file' is super nice and flexible at first, but either makes stuff very difficult to change (the file becomes a poorly-specified inconvenient ad-hoc API), or makes it very hard to build applications that work in the long term (because the application has to write a change-tolerant parser). It makes a very cool demo (look, I can do stuff to TCP sockets using bash!), and really convenient for ad-hoc stuff, but is inferior to a real API for applications that need to keep working year after year after year while still being able to patch the rest of the system.
The file tree is just a way of discovering endpoints to the API. You still need an encoding to read/write to those endpoints.
The same is true over a "real" API with one big difference: there's not a small set of syscalls to access the endpoints and read/write binary data to them. Thats more stable than any FFI or encoding scheme.
I'll take "everything is a file" over "everything is an object that has a special mechanism for querying interfaces and dispatching to handlers" or "everything is RPC" or "everything is REST" because it has the smallest surface area.
>The same is true over a "real" API with one big difference: there's not a small set of syscalls to access the endpoints and read/write binary data to them.
What do you mean? Binder from Android uses just the ioctl syscall.
>because it has the smallest surface area.
I assure you the 1 syscalls for using binder (unless you count open and close of /dev/binder) is less syscalls than what it takes when everything is a file (read, write, ioctl, lseek, lstat, etc).
It's the job of the operating system to provide a stable foreign function interface, and dynamically linked libraries to access common operating system apis that aren't covered by system calls.
Smallest surface area??! There's no excuse for Unix forcing you to use the fractally complex sucking chest wound of ioctl and brittle binary raw pointerless C structures in memory, instead of simply calling functions and passing normal parameters and pointers and results in and out.
Computers are great at and have a long history of calling functions, so why not just support that, huh?
"Everything is a file" is pointlessly performative but hopelessly incomplete Bauhaus Minimalism that fails to reduce but instead multiplies and relocates all the complexity to the Brutalist Rawness of ioctl's uncanny leaky abstraction of a kitchen sink, which undermines the entire Unix presumptuous philosophy of simplicity and consistency.
I think the big problem with objects is that they're just not the common denominator: e.g., image files/formats are going to be byte streams.
That, and getting the world to agree on object formats is like pulling teeth, it seems. E.g., WASM just wanted to add a string type, which OUGHT TO¹ be simple (use UTF-8) … but alas.
JSON? Not enough types. YAML? Doesn't matter if it's good, nobody likes YAML. Etc.
And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network.
I start thinking "Perhaps something could be built on top of byte streams" but that's YAML, JSON, CBOR, take your pick.
I don't understand. Objects existing doesn't mean files can no longer exist.
>getting the world to agree on object formats is like pulling teeth
In Windows apps agree, Mac app agree, iOS app agree, Android apps agree, Java apps agree. Sending or receiving strings to a system API is common.
>And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network.
Yes, but I'd much prefer if there was the 1 true way an os chooses it to be than each thing inventing its own way.
Well, there's dbus. On the other hand if you want a function and not protocol, why would you care what protocol it uses? It might as well be a file. When you call getaddrinfo("localhost"), it reads /etc/hosts - a text file.
Even PowerShell's flawed and limited execution definitively proves how terribly misguided and morally bankrupt the pin-headed one-dimensional "everything is a file" Unix philosophy is, especially with its the ridiculously hard-to-parse ad-hoc non-standard mixed-bag syntax of /proc and /etc files, which would have been much better simply using xml or json, yet which don't even come close to achieving that low bar of machine readability and ease of scripting.
If everything is a file, then why the fuck is the insufferable abomination of ioctl so crucial to haphazardly papier-mâché and duct tape over all the uncanny gaps and gaping cracks?
Yet another strike against the mindless cargo cult nonsense and unquestioning sycophantic hero worship of TAOUP.
Sure It Corrupts Your Files,
But Look How Fast It Is!
Pretty daring of you to be storing important files on a Unix system.
—Robert E. Seastrom
The traditional Unix file system is a grotesque hack that, over the years,
has been enshrined as a “standard” by virtue of its widespread use. Indeed,
after years of indoctrination and brainwashing, people now accept Unix’s
flaws as desired features. It’s like a cancer victim’s immune system
enshrining the carcinoma cell as ideal because the body is so good at making them.
Way back in the chapter “Welcome, New User” we started a list of what’s
wrong with the Unix file systems. For users, we wrote, the the most obvious failing is that the file systems don’t have version numbers and Unix
doesn’t have an “undelete” capability—two faults that combine like
sodium and water in the hands of most users.
But the real faults of Unix file systems run far deeper than these two missing features. The faults are not faults of execution, but of ideology. With
Unix, we often are told that “everything is a file.” Thus, it’s not surprising
that many of Unix’s fundamental faults lie with the file system as well.
But I also think there's a deeper set of problems in the "genetics" of Unix, in that it supports a "reductive" form of problem solving, but doesn't help at all if you want to build abstractions. Let's say one of the core ideas in Unix is "everything is a file", i.e., read/write/seek/etc. is the universal interface across devices, files, pipes, etc.). "Everything is a file" insulates a program from some (but not all!) irrelevant details of the mechanics of moving bytes into and out of RAM... by forcing all programs to contend with even more profoundly irrelevant details about how those bytes in RAM should be interpreted as data in the program! While it is sometimes useful to be able to peek or poke at bits in stray spots, most programs implicitly or explicitly traffic in in data relevant to that program. While every such datum must be /realized/ as bytes somewhere, operating on some datum's realization /as bytes/ (or by convention, as text) is mostly a place to make mistakes.
Here's an example: consider the question "who uses bash as their login shell?" A classical "Unixy" methodology to attacking such a problem is supposed to be to (a) figure out how to get a byte stream containing the information you want, and then (b) figure out how to apply some tools to extract and perhaps transform that stream into the desired stream. So maybe you know that /etc/passwd one way to get that strea...
>So maybe you know that /etc/passwd one way to get that stream on your system, and you decide to use awk for this problem, and type
>awk -F: '$6 ~ "bash$" { print $1 }' /etc/passwd
I like to apply similar logic to when I use search engines and start broad, narrowing down as needed. If you jump straight to grep or awk you may entirely miss what you were looking for. I frequently search my local IRC logs, and my tool of choice is `less`. I would probably also use that here. You can comfortably get around, all the data is there, and you can search within it. You see adjacent lines which may be relevant to your interests.
You could maybe describe this difference in mindset as programmers vs users. I consider myself a user, but not a programmer. Just as in your example, I thought to look in /etc/passwd for the desired info, but I don't understand why you jumped to that gnarly awk one-liner. Maybe the missing context is you were writing some sort of script/program, but the way you framed it read to me like "I want to get this information on a unix-like machine", and as a human I would just open the file in less. Using cat is another option, but I hate when a file was longer than expected and I have to scroll way up after printing it out. Plus, it's easier to search the text with less than it is in my terminal emulator (though tmux scrollback search helps in a pinch).
I find that the unix approach to things often feels intuitive from a user-perspective where the cranky programmers who insist using python would be better than working with shell have some alternative solution I can't really grok. Maybe it's just people preferring what they're used to, though.
You are somehow right but miss some points out of anger
For instance, your awk example with /etc/passwd : the important thing is not that awk fails to be the perfect way to parse the file's content. The important thing is that you can parse the file with a generic tool like awk.
To me, this is the good idea behind all this : using a standard set of tool, you can process the data.
I cannot find a better alternative, and you failed to provide one too : standardize the files ? Their format, yes, not their content. Create a dedicated tool for each kind of file ? Good idea, people would need to learn hundreds of tools, one for each file.
Because most of those files are stored in ascii and can be handled as regular files, you can use just cat to get the content, you do not need to run a specific tool with internal binary parser that create syscall or grab data from shared memory or whatever (but you could).
Another thing : a file-system with an "undelete" capability is a filesystem that do not support data deletion. That would be pretty useless for most use cases, so most filesystems (from the Linux's gang to ntfs to ufs to refs to fat) do not implement this.
A /proc-like filesystem could use binary formats that would have better security (no formatted printing and scanning bugs) as well as performance. (While not to mention having disadvantages also.)
The article somewhat conflates "everything is a file" with "everything is a text file".
Also, "everything is a file descriptor" is usefully different from "everything is a file". Not every resource bound to a file descriptor is a file (in the sense that it was obtained from a name resolution in the filesystem).
Things that are not files, but part of "everything is a file descriptor": socket(), socketpair(), pipe().
Both pipes and sockets are files, in the sense that they can appear on the filesystem. Some people refer to pipe(2) pipes as anonymous pipes — they're anonymous because they're not in the filesystem. (One in there is a "named" pipe.)
(I do sort of wish most object creation in the FS was a pair of,
int fd = create_a_file_descriptor_of_some_type();
link_fd(fd, "/the/name/for/it");
I think that'd be a cleaner abstraction, and then introduce syscalls on top of that for doing them together in case you think that's too much user/kernel traffic.)
Linux does have the O_TMPFILE flag to open. This creates a file with no name. But importantly it still needs a directory path because it needs to know which filesystem the data should be stored on. With your example if you wrote a few gigs of data to that fd before linking it the kernel would need to either buffer it (maybe swapping it out) or guess and then copy the data when it guesses wrong.
Aside: why does this have a wikipedia article? A Unix concept like this should simply be a section on the base Unix article no? Surprised the editors didn't squash it
The article is about 12 years old. Unix is one of those topics on Wikipedia where experts wrote a lot of trivial articles, for every shell builtin and implementation details. Gradually, these are challenged for independent notability and merged into larger articles, or deemed to be no more than dictionary definitions.
There are some interesting points raised on the Talk page: that this alleged philosophy wasn't really espoused or coined by the implementors, so there's one angle for you.
23 comments
[ 3.1 ms ] story [ 77.8 ms ] thread> and therefore a more accurate description of this feature is Everything is a file descriptor
The filesystem is where files go to get names, and to be organized. But critically, not everything is a normal file, which I think is where people experience the most "the abstraction is leaky!" feelings. I don't think the abstraction needs everything to be a normal file, but I think there is still a lot of use in having a unified namespace for all the various types of files, normal or otherwise. (As opposed to having disparate namespaces for files of different types.)
For the "file descriptor" half, the value is being able to wait on events/event completion for a set of heterogeneous file descriptors, but boy did it take a while to get there with usable interfaces for that.
I would rather there be a proper API for remote objects rather then every remote object having to roll its own protocol for doing the same.
We have socket, connect|{bind, listen, accept}, read, write.
For connectionless, send, recv.
Sockets are not part of "everything is a file" but of "everything is a file descriptor".
I disagree. I should be able to just call a function. I don't want to bother figuring out the network protocol that the process decided on. It's too low level to expose.
The same is true over a "real" API with one big difference: there's not a small set of syscalls to access the endpoints and read/write binary data to them. Thats more stable than any FFI or encoding scheme.
I'll take "everything is a file" over "everything is an object that has a special mechanism for querying interfaces and dispatching to handlers" or "everything is RPC" or "everything is REST" because it has the smallest surface area.
What do you mean? Binder from Android uses just the ioctl syscall.
>because it has the smallest surface area.
I assure you the 1 syscalls for using binder (unless you count open and close of /dev/binder) is less syscalls than what it takes when everything is a file (read, write, ioctl, lseek, lstat, etc).
Smallest surface area??! There's no excuse for Unix forcing you to use the fractally complex sucking chest wound of ioctl and brittle binary raw pointerless C structures in memory, instead of simply calling functions and passing normal parameters and pointers and results in and out.
Computers are great at and have a long history of calling functions, so why not just support that, huh?
"Everything is a file" is pointlessly performative but hopelessly incomplete Bauhaus Minimalism that fails to reduce but instead multiplies and relocates all the complexity to the Brutalist Rawness of ioctl's uncanny leaky abstraction of a kitchen sink, which undermines the entire Unix presumptuous philosophy of simplicity and consistency.
That, and getting the world to agree on object formats is like pulling teeth, it seems. E.g., WASM just wanted to add a string type, which OUGHT TO¹ be simple (use UTF-8) … but alas.
JSON? Not enough types. YAML? Doesn't matter if it's good, nobody likes YAML. Etc.
And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network.
I start thinking "Perhaps something could be built on top of byte streams" but that's YAML, JSON, CBOR, take your pick.
¹(this is an RFC 6919 OUGHT TO)
>getting the world to agree on object formats is like pulling teeth
In Windows apps agree, Mac app agree, iOS app agree, Android apps agree, Java apps agree. Sending or receiving strings to a system API is common.
>And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network.
Yes, but I'd much prefer if there was the 1 true way an os chooses it to be than each thing inventing its own way.
If everything is a file, then why the fuck is the insufferable abomination of ioctl so crucial to haphazardly papier-mâché and duct tape over all the uncanny gaps and gaping cracks?
Yet another strike against the mindless cargo cult nonsense and unquestioning sycophantic hero worship of TAOUP.
https://web.mit.edu/~simsong/www/ugh.pdf
13 The File System
Sure It Corrupts Your Files, But Look How Fast It Is!
Pretty daring of you to be storing important files on a Unix system. —Robert E. Seastrom
The traditional Unix file system is a grotesque hack that, over the years, has been enshrined as a “standard” by virtue of its widespread use. Indeed, after years of indoctrination and brainwashing, people now accept Unix’s flaws as desired features. It’s like a cancer victim’s immune system enshrining the carcinoma cell as ideal because the body is so good at making them.
Way back in the chapter “Welcome, New User” we started a list of what’s wrong with the Unix file systems. For users, we wrote, the the most obvious failing is that the file systems don’t have version numbers and Unix doesn’t have an “undelete” capability—two faults that combine like sodium and water in the hands of most users.
But the real faults of Unix file systems run far deeper than these two missing features. The faults are not faults of execution, but of ideology. With Unix, we often are told that “everything is a file.” Thus, it’s not surprising that many of Unix’s fundamental faults lie with the file system as well.
https://news.ycombinator.com/item?id=26604833
Ask HN: What are the bad parts of Unix?
https://news.ycombinator.com/item?id=26613607
lapsed_lisper on March 28, 2021 | prev | next [–]
[...]
But I also think there's a deeper set of problems in the "genetics" of Unix, in that it supports a "reductive" form of problem solving, but doesn't help at all if you want to build abstractions. Let's say one of the core ideas in Unix is "everything is a file", i.e., read/write/seek/etc. is the universal interface across devices, files, pipes, etc.). "Everything is a file" insulates a program from some (but not all!) irrelevant details of the mechanics of moving bytes into and out of RAM... by forcing all programs to contend with even more profoundly irrelevant details about how those bytes in RAM should be interpreted as data in the program! While it is sometimes useful to be able to peek or poke at bits in stray spots, most programs implicitly or explicitly traffic in in data relevant to that program. While every such datum must be /realized/ as bytes somewhere, operating on some datum's realization /as bytes/ (or by convention, as text) is mostly a place to make mistakes.
Here's an example: consider the question "who uses bash as their login shell?" A classical "Unixy" methodology to attacking such a problem is supposed to be to (a) figure out how to get a byte stream containing the information you want, and then (b) figure out how to apply some tools to extract and perhaps transform that stream into the desired stream. So maybe you know that /etc/passwd one way to get that strea...
>awk -F: '$6 ~ "bash$" { print $1 }' /etc/passwd
I like to apply similar logic to when I use search engines and start broad, narrowing down as needed. If you jump straight to grep or awk you may entirely miss what you were looking for. I frequently search my local IRC logs, and my tool of choice is `less`. I would probably also use that here. You can comfortably get around, all the data is there, and you can search within it. You see adjacent lines which may be relevant to your interests.
You could maybe describe this difference in mindset as programmers vs users. I consider myself a user, but not a programmer. Just as in your example, I thought to look in /etc/passwd for the desired info, but I don't understand why you jumped to that gnarly awk one-liner. Maybe the missing context is you were writing some sort of script/program, but the way you framed it read to me like "I want to get this information on a unix-like machine", and as a human I would just open the file in less. Using cat is another option, but I hate when a file was longer than expected and I have to scroll way up after printing it out. Plus, it's easier to search the text with less than it is in my terminal emulator (though tmux scrollback search helps in a pinch).
I find that the unix approach to things often feels intuitive from a user-perspective where the cranky programmers who insist using python would be better than working with shell have some alternative solution I can't really grok. Maybe it's just people preferring what they're used to, though.
For instance, your awk example with /etc/passwd : the important thing is not that awk fails to be the perfect way to parse the file's content. The important thing is that you can parse the file with a generic tool like awk.
To me, this is the good idea behind all this : using a standard set of tool, you can process the data.
I cannot find a better alternative, and you failed to provide one too : standardize the files ? Their format, yes, not their content. Create a dedicated tool for each kind of file ? Good idea, people would need to learn hundreds of tools, one for each file.
Because most of those files are stored in ascii and can be handled as regular files, you can use just cat to get the content, you do not need to run a specific tool with internal binary parser that create syscall or grab data from shared memory or whatever (but you could).
Another thing : a file-system with an "undelete" capability is a filesystem that do not support data deletion. That would be pretty useless for most use cases, so most filesystems (from the Linux's gang to ntfs to ufs to refs to fat) do not implement this.
A /proc-like filesystem could use binary formats that would have better security (no formatted printing and scanning bugs) as well as performance. (While not to mention having disadvantages also.)
The article somewhat conflates "everything is a file" with "everything is a text file".
Also, "everything is a file descriptor" is usefully different from "everything is a file". Not every resource bound to a file descriptor is a file (in the sense that it was obtained from a name resolution in the filesystem).
Things that are not files, but part of "everything is a file descriptor": socket(), socketpair(), pipe().
(I do sort of wish most object creation in the FS was a pair of,
I think that'd be a cleaner abstraction, and then introduce syscalls on top of that for doing them together in case you think that's too much user/kernel traffic.)There are some interesting points raised on the Talk page: that this alleged philosophy wasn't really espoused or coined by the implementors, so there's one angle for you.