It's probably worth the owner checking out the Lisp Machine, Oberon, and Inferon (A descendent of Plan9 where they improved over the original concept). They say Plan9 is a step in the right direction, but rather than attacking a specific aspect of Plan9, they critique how the concepts were imported into Linux, which seems inherently silly to me.
It's like critiquing a master craftman's tools by observing how a master craftman uses his tools, and then critiquing how the apprentice uses those tools. Sure, some of that criticism of the concepts is valid, but they picked the worst implementation of those concepts to critique.
One of the things mentioned in the critique is how the filesystem in question communicates the changes back to the user, and how the user is expected to know what parts of the filesystem to alter. While in the examples given, it does seem relatively obscure, the decry the fact that they had to check the manual for the position to write to.
However, both of those would still be the case for the system that they talk about. Let's talk about object orientated systems, which already exist, in limited (compared to the operating system) forms. You still have to pick up a manual to figure out what keys and functions are relevant to you. In Pharo (which is more or less the system they're after!), you can inspect objects that exist and the methods that exist for them -- however -- often, similarily-named functions do different things, which also requires reading a manual. In addition, using these tools as a first-time user, I was overwhelmed by the number of functions available, most of which I could only guess at what their purpose was. Pharo integrates the manual into the system, but the manual is still there!
Standardized error codes are given in structures passed back (As in Erlang) or in Exceptions, but there are problems with those too. You still have to figure out what those exceptions or errors mean, and there isn't necessarily a standardized format for those errors. As a developer I've been recently been working on writing a Python api. As someone who has little experience with the intimate details of the Python language, I do not always know of the exceptions that exist that I am able to throw, I do not always know of which exceptions would be most appropriate. The same problem exists (obscure error values), but in a different form. There are existing libraries that I rely on that implement more exceptions, sometimes these aren't documented, and even with the venerable Requests library, I have still had to crack open the objects that exist to find easier methods of passing data that the Requests library has, and uses, but does not document for external use.
Let's look at the windows hive database. That's a database of keys for values in the operating system. As a random developer, would you be able to open it and figure out what it does? I wouldn't. As a windows power user I often relyied on articles from howtogeek without really fully understanding what the keys were doing (Although because of my experience an a systems developer, I could guess, but only after the fact). Again, the same problem ("I know what I want to do, but I do not know how to do it") is exposed in a different form, and the methods and practices of the Microsoft organization make that difficult hard to reach. Yet again, the same problems are there, but in a different form.
I do agree that the shell should handle program arguments, a program could expose keys and values and a shell could read that and tell the user about it. I would be interested to see the problems that the Oil shell encounter in their implementation of an object-orientated shell.
> They say Plan9 is a step in the right direction, but rather than attacking a specific aspect of Plan9, they critique how the concepts were imported into Linux, which seems inherently silly to me.
Well, I've played with Plan9 several times, but I don't really feel like criticizing it, because I don't really know it that much. Only thing worth of criticizing is that it almost feels like objects, but not implemented fully.
I agree, that criticism of Plan9 would make more sense from philosophical point of view, but it wouldn't be authentic.
> However, both of those would still be the case for the system that they talk about. Let's talk about object orientated systems, which already exist, in limited (compared to the operating system) forms. You still have to pick up a manual to figure out what keys and functions are relevant to you. In Pharo (which is more or less the system they're after!), you can inspect objects that exist and the methods that exist for them -- however -- often, similarily-named functions do different things, which also requires reading a manual. In addition, using these tools as a first-time user, I was overwhelmed by the number of functions available, most of which I could only guess at what their purpose was. Pharo integrates the manual into the system, but the manual is still there!
Thats true, but you can also do a type-checking and infer a lot only from the proper naming of the objects and methods in the "Clean code" (book) style.
> Let's look at the windows hive database. That's a database of keys for values in the operating system. As a random developer, would you be able to open it and figure out what it does? I wouldn't.
One of the things that is not strongly captured in this article and which I since consider increasingly important is ability to use reflection on the system. Database that you can't open and figure out what it does is not worth using.
I get what the programmer is trying to say but I think the problem is when you think everything is a nail because all you have is a OOP hammer. OOP lets you wire things up in a very unnatural memory hungry way kinda way.
When you start dealing with 500,000 files in one directory and you try to do a files->getname(10)->size->to_string(). You are bound to run out of physical memory. This is probably why newer operating systems are so slow. Caching will only save you for so long.
I don't actually really like the concept of the OOP as it is understood today. I mean it more in the Self / Smalltalk sense, which is quite different and thanks to late binding more similar to lisp.
Also your example is weird and you won't run out of memory if you use generators.
Generators are OOP service packs. It feels like the intention is to make things simple and easy to mesh together at the cost of overall flexibility. This would make a case for a new operating system built around this idea.
Well, its just criticism. I am working on my own programming language (http://blog.rfox.eu/en/tinySelf.html) which could be probably used for experiments on this field of work, but mostly I am focused on creating personal wiki systems these days.
interesting and confusing language. if we really want to get new break throughs we will eventually have to tackle the root cause of issues and re-write the OS or we will be patching languages forever. For example; image every variable would be data + meta? It would take up twice the space but allow for stranger types.
General purpose editing of structured data. Org-mode is the closest thing I know of but it is still infuriatingly ad-hoc and text-first.
Grepping (search in) and diff/patch of structured data. While many text manipulations are easy to implement in search/replace manner, structured data are more involved.
Errors and other kinds of unexpected input/output. Semantics of unix process failure are clear. State of the system after one process crashes is readily known. We can compare the filesystem before/after.
If we instead throw an exception in object-oriented runtime, the result is not so easy to analyze or compare, there may be dangling references or other complications. I don't know of any language runtime that can dump itself whole in (semi-)readable format for later comparisons.
I suspect that one of the keys to user-friendly structured data will be storing sparse data, in the sense of incompletely defined data (note that's real persistence, not just existing in the editor). It's also important to allow data that doesn't (yet) conform to whatever schema is relevant, but I think sparse data will generalize to that case.
One of the undersold features of plain text is being able to exist in "invalid" states while it's being edited. Structured data UI needs to have a least that level of malleability to have a chance over the current paradigm.
Definitely don't do it that way, then. Give comments first class status.
Comments should actually be independent of the underlying structure they're documenting. I'd like them to be an independent layer of annotations that can be applied to any data.
One reason that we don't "just stored structured data as-is" is because there are many kinds of structured data where the in-memory representation is architecture and/or compiler specific.
Many a novice programmer has no doubt made the mistake of thinking that you could, for instance, do this to deliver a C/C++ struct across a socket:
that works until the reader and writer are on machines with different architectures.
Same observation could be made about filesystems used to store "structured data". We serialize and deserialize it because the in-memory representation is not inherently portable.
The author's point is that if our APIs worked with structured data instead of byte streams, then we wouldn't need to care about the in-memory format. The standard API would do the hard work for us, allowing us to call send(fd, my_struct);
Since there's no standard API (blessed by the OS) and the lowest common denominator is byte streams, we're seeing all these ad-hoc solutions and a hodgepodge of formats and libraries to deal with. That's lots and lots of time and money spent on rather basic stuff.
And that's why I think the author is wrong about this (he is probably right about thinking (more often) about filesystems as databases, but that's somewhat orthogonal).
The approach you're describing only works for POD-style "structured data". Once you start using OOP of almost any type (though not every type), you no longer have ... well, POD that you can move to/from a storage medium. You have objects whose in-memory format IS important and compiler dependent.
There are other concerns too. His WAV example (I write pro-audio software for a living) doesn't even begin to touch on the actual complexities of dealing with substantial quantities of audio (or any other data that cannot reasonably be expected to fit into memory). Nor on the cost of changing the data ... does the entire data set need to be rewritten, or just part of it? How would you know which sections matter? Oh wait, the data fundamentally IS a byte stream, so now you have to treat it like that. If you don't care about performance (or storage overhead, but that's less and less of a concern these days), there are all kinds of ways of hiding these sorts of details. But the moment that performance starts to matter, you need to get back the bytestream level.
And so yes, there's no standard API and yes the lowest common denominator is byte streams ... because the __only__ common denominator is byte streams. Thinking about this any other way is a repeat of a somewhat typical geek dream that the world (or a representation of the world or part of it) can be completely ordered and part of a universal system.
Of course it can be streamed! Nobody ever suggested it could not be. The point is that to stream it portably (i.e. without knowing the hardware characteristics - and possibly software characteristics too - of the reciever) you have to first serialize it and then deserialize it, because the in-memory representation within the sender is NOT portable.
You're too hung up on in-memory representation. Yes, if it's not right, then it needs to be converted. That can be done for you, or you can do it manually with byte streams like cave man. If you can do it manually fast, then it can be done just as fast automatically based on the declared structure.
It isn't possible to be in a world where the in-memory representation doesn't matter. Someone has to write and maintain that code, and if it's you, then you get the fun task of telling some users their workload won't fit into it. And then they go off and write their own in-memory representation.
As I’ve noted above, this problem would go away if data streams were adequately tagged in the first place.
Having that high-level knowledge of data structure enables all sorts of intelligent automation.
In the event that the client uses a different memory layout, it could look up a coercion handler that converts the supplied data from its original layout to the layout required by the client. This is, for instance, how the Apple Event Manager was designed to work: all data is tagged with a type descriptor:
and so on. (The tags themselves are encoded as UInt32; nothing so advanced as MIME types, but at least they’re compact.)
The AEM includes a number of standard coercion handlers for converting data between different representations, and clients may also supply their own handlers if needed. Thus the server just packs data in its current representation, and if the client uses the same representation then, great, it can use it as-is. Otherwise the client-side AEM automatically coerces the data to the form the client as part of the unpacking process.
There are limits in the AEM’s design, not least the inability to describe complex data (arrays and structs) with a single “generic” descriptor, e.g. `AEList(SInt32)`. That would vastly simplify packing and unpacking—in best case to simple flat serialization/deserialization, at worst to a single recursive deserialization—instead of two recursive operations with lots of extra mallocs and frees for interim data. But the basic principle is sound, and adheres well to the “be liberal in what you accept” principle.
I believe Powershell does something similar when connecting outputs to inputs of different (though broadly compatible) types, intelligently coercing the output data to the exact type the input requires. No manual work required; it “Just Works”.
Or, if you don’t mind the extra overhead then content negotiation is also an option, which is something HTTP does very well (though web programmers very badly). That is advantageous when communicating with “less intelligent” clients as it permits the server, which best understands its own data, to pre-convert (e.g. via lossy coercion) its data to a form the client will accept.
Lots of ways that Unix’s “throw its hands up and dump the problem all over the users” non-answer can be massively improved on, in other words, without ever losing the lovely loose coupling that is a Unix system’s strength. It only requires a single piece of essential—yet missing—information: the data’s type.
The problem outlined in the original article isn't about data streams. It is, at bottom, about the contrast between data storage and in-memory representation.
Typed data streams were not invented by Apple. Back in the 1980s, there was (for example) Sun's RPC mechanism, which gave you "seamless" remote procedure call, including transfer of arbitrary structures over a network.
But the original post is much more about filesystems. I used the socket example merely to illustrate the problem, not the actual topic.
I agree especially with the proposition that we should exchange structured data (via files, network connections, ...).
I started this project several years ago to support live code updates in production systems without sacrificing performance, but I eventually hit the same issues with exchanging structured data:
https://github.com/Morgan-Stanley/hobbes
I split out structured logging into a single header-only lib, structured RPC into a single header-only lib, etc. It's not quite the same as having native support in the OS, but because the mechanism is so lightweight (don't even need to link anything special), it's a pretty good substitute.
Preach! I've felt this way too, that adding structured data as a universal feature to operating systems would be a pretty agreeable next step.
I wonder if we're past the point of return, though, in terms of technical divergence. It sounds like, in the Ancient Times, there was a handful of great programmers whose work created the world we program in now. But now, there's way more programmers being paid to make slightly different versions of this "next step", and it would require widespread agreement/coordination to implement it on a scale where it's a seamless feature that's taken for granted the way the shell/network/fs are.
Article resume: biggest cs problem now is a diversity of serialization formats, because most of current code consist of parsing various formats, so OS must to do something about it.
No, it is not a so big problem. And no it will not do our life easier. Also author did not mention about the real problem of semantic. How client should interpret a structure to compose a valid request.
OS should not know about userspace structures because OS don't do anything with it. It stores and transfers chunks of bytes and its semantic is defined by userspace. And forcing current popular serializing format on OS level is the most dumb idea ever.
most dumb idea ever is a bit strong, but yeah. If you could get OSs to adopt this, then as an app writer, you're going to have to worry about how Microsoft's jacked up version of the standard broke your content when it was moved between computers or even OS versions. You'd have the UNIX/MS line ending problem not just in text files, but with every Type recognized by the database
>And forcing current popular serializing format on OS level is the most dumb idea ever.
The idea is that you take the common elements of all of those serialization formats and when you take a good look you notice that the lowest common denominator isn't actually raw bytes on a disk.
I think the browser is the described OS here: every-api passes objects around, IndexDB is your filesystem, remote REST & GraphQL servers are returning structured JSON objects to your program, you can add a <script> tag to your <html> to pull in your single-function scripts (i.e. leftpad).
Seriously though, the question of "why isn't it done this way" is answered a couple times: "it's all just bytes in the end". Bytes & arrays-of-bytes are really the only thing that you can trust that all systems and all languages interpret the same (even 2-byte integers can be interpreted differently by different systems). Presumably the author just wants to be able to memcpy structures between programs or from disk, or maybe with some high efficiency pre-allocated heap stuff, but you really need to have code to validate the bytes, essentially requiring the parsing of your data-structure which the author is trying to avoid.
Running programs with some common argument notation does sound really nice, but ideally that's just calling a function from your language's binding to their library (as their executable should do, right?).
> remote REST & GraphQL servers are returning structured JSON objects to your program
They are not. They are sending a HTTP message, which consists of just octets and has to be parsed. The parsers have to have a ridiculous complexity for various reasons. The HTTP header contains sublanguages such as RFC 7231 §5.3.1 or RFC 8288 §3.
JSON meanwhile has amassed at least six incompatible specifications with the same exact media type, so there is not even a theoretic hope of parsing all messages correctly. https://news.ycombinator.com/item?id=20736665
It links to several RFCs. RFC-7158 and RFC-7159 differ only in (a) that one is RFC-7158 and the other is RFC-7159 (b) that one is dated in March 2013 and the other is March 2014 (c) that one obsoletes RFC-7158 in addition to RFC-4627 which they both obsolete. It appears that 7159 was pushed only to handle an error in the dating of 7158, given that the irrelevant RFC-7157 is dated March 2014.
So to say that there's more that five incompatible specs is just a cruel disregard of the facts. No implementation can differ in its parsing of a text based on the metadata of the spec. There needs to be a substantive difference in the spec.
As to the current RFC 8259, it makes no substantial differences from the previous version. There are a few typos that have been fixed ("though" becoming "through", when "though" was not a possible interpretation). It eliminates the obligation to parse public texts not coded in UTF-8. Previously, documents SHALL be UTF-8 - that's barely a change, but it is a change. It has specified more concretely its relationship with ECMA-404. It's possible that someone reading earlier RFCs would have concluded "a valid RFC 7159 json parser can parse and only the documents produced by ECMA-404". If there were any discrepancies between ECMA-404 and RFC-7159, therefore, an interpretation along these lines would lead a person to conclude you cannot parse a document according to RFC-7159, so the spec doesn't really count as an incompatible spec. Under the current spec, it identifies that there are possible discrepancies, and RFC-8259 is intended to accept fewer documents that ECMA-404.
The first RFC did not permit the json documents "true", "false" and "null", accepting only json documents that comprise an object or an array. It permitted any Unicode encoding, defaulting to UTF-8, and non-Unicode encodings as well (so a Latin-1 document is valid RFC-4627, but not valid RFC-7158+). It also included some incorrect statements about its relationship to javascript. These were reduced in later versions and eventually eliminated. They do not affect the specification of the language, but merely how you may handle data in the language.
No document that is accepted by the current RFC will be rejected by any RFC other than the first. The only such documents to be rejected by the first are those which consist entirely of "true", "false", or "null". No UTF-8 document that is accepted by the first RFC will be rejected by subsequent RFC parsers, unless a person reads a paragraph called "security considerations" that says "you can kinda do this but it's insecure" as somehow trumping the clear statements of the grammar in earlier sections.
I have not investigated the other specs and I probably won't. But the idea that there are 6 incompatible specs is false. There is not more than 5, and it is almost trivial to accept documents according to 3 of 5 or 4 of 6 specs at once by ignoring the restrictions on character coding.
> It links to several RFCs. RFC-7158 and RFC-7159 differ only in (a) that one is RFC-7158 and the other is RFC-7159 (b) that one is dated in March 2013 and the other is March 2014 (c) that one obsoletes RFC-7158 in addition to RFC-4627 which they both obsolete. It appears that 7159 was pushed only to handle an error in the dating of 7158, given that the irrelevant RFC-7157 is dated March 2014.
"they only differ in", yeah, indeed.
Dude. We all have work to do and grappling with some obscure JSON parsing corner is the last thing we need in our workday. This applies to 95% of commercial programmers, I am willing to bet.
What you described does sound simple and small in isolation. Now multiply it by 50 and see how "they only differ in" is a problem that must not be allowed in the first place.
Where's the-one-and-only JSON spec? Why no organisation has the courage to step forward and standardise it once and for all?
Given the current realities, just cut your very small and ignorable losses by losing those obscure JSON parsing corners and let people describe JSON simply as:
"RFC #ABCD".
That's it. Nothing else. Only that. Everything is contained in it.
Complexity compounds and makes everything non-deterministic. We should stop allowing that.
Yes but that's the underlying system, not your "application code" handling the structure parsing. Maybe I shouldn't have said JSON, but you do always get a JavaScript object back, as opposed to UNIX filesystem/socket operations which only really return application/x-octet-stream data. I thought that was the thesis of the article: it'd be a better world if your operating system supported parsing common structures for you.
i think android & iOS provide the data structures the author is looking for e.g. queues scheduled tasks, relational storage, key value and mostly typesafe
New problem: Data serialization is now centralized and there are N ways of converting between the centralized way it's been serialized and everyone's incompatible structural needs.
Hey, at least then we've improved the format conversion problem from O(N^2) to O(N). :)
In practice, though, you end up with something like XML, which (to steal a phrase, I wish I could remember where I read it) is less a common language than it is an alphabet. I guess it's better than CSV.
A lot of these criticisms are about the OS not doing enough high level stuff for the programmer. If you want to see what happens when the OS does all that, have a look at Android, where the "OS" is what's giving you sockets, key value stores, structured logging, etc. because the only real APIs it's giving you are the Java ones. Now look at the horrible workarounds you need to apply to make old code compatible with newer versions as demands changed over the years tk focus more on user privacy and battery life.
For nearly every problem encountered here there's a standard people just ignore. You can implement structured logging at OS level, like Windows does, and see random binary data drops and text files appear all over your logging interface and your file system. Linux config should follow the XDG standard so that configuration for applications can be structured into directories except half the programs don't do that. Windows provides an API for this stuff (the registry) and it's near impossible to configure an application using the registry. The registry is also combined with random config files all over the system, of course, to make it extra difficult to modify a program's behaviour.
Most nice modern features are also missing because operating systems are still being made in C, not C++. The OS doesn't have a concept of objects so yes your socket function will have to fall back to calling select(). A lot of these abstractions for basic communications have been solved on Linux using DBUS, which provides a somewhat standardised interface for many OS daemon and GUI features, all not being used because programmers forget that it exists or because programmers want to use their own solution instead.
On Windows there's COM to try and help with that and well, see where that ended up: a versioned mess of pointers and factories to try to make it easier for everyone, where functionality sometimes completely breaks or needs to be emulated because it turned out the high level concept had a design flaw and now programs won't work if you fix it.
I've done some thought experiments about structuring an OS and a file system to store data consistently and easily parsable, with modern bindings for most concepts. In the end I've had to conclude that the only way to keep the system working like intended would be to either convince everyone to do exactly as I say or to only allow me to write software for such a system. Whatever structure I can think off will inevitably be too constrained for someone else and the middleware abstraction problem starts all over again.
My only conclusion is that I want the OS to be as simple as possible with people following common standards when they write applications, such as using YAML/ini/XML configuration with Syslogd logging and XDG directory structures for user data, with the technical abstractions left at the library level. If we can just get that, most of the inconsistency problem would be solved, but even this is too difficult to do in practice as it turns out.
I understand the natural frustrations articulated here, especially given OP’s experience working with files, but it seems to dismiss what is actually a core strength of current operating systems: they work. Given a program supporting 16 bit address spaces from the 1970s, you can load it into a modern x86 OS today and it works. This is an incredible feat and one that deserves a lot more recognition than offered here! Throughout an exponential explosion of complexity in computing systems since the 70s, every rational effort has been made to preserve compatibility.
The system outlined here seems to purposefully avoid it! Some sort of ACID compliant database analogy to a filesystem sounds nice until 20 years down the line when ACIDXYZABC2.4 is released and you have to bend over backwards to remain compatible. Or until Windows has a bug in their OS-native YAML parser (as suggested here) so now your program doesn’t work on Windows until they patch it. But when they do, oh no you can’t just tell your users to download a new binary. Now they have to upgrade their whole OS! Absolute chaos. And if you’re betting on the longevity of YAML/JSON over binary data, well just look at XML.
Want to admire your fancy After Dark win 3.1 screensaver? Just emulate the whole environment! We don't want to keep suporting the broken architectures and leaky abstractions of past, they drag us down. Microsoft's dedication to backwards compatibility is admirable but IMO misguided and unsustainable in the long run. The IT industry has huge problem with complexity. We need to simplify the whole computing stack in interest of reliability, security and future innovations.
The proposed improvement as I understood it would be future proof. It seems trivial to build a rock-solid YAML/XML/JSON/EDN parser on OS level, and since it would be so crucial part of OS the mistakes would be caught and fixed quickly. It shouldn't even matter if structured data syntax is replaced or expanded in future, as long as it is versioned and never redacted. Rich Hickey's talk "Spec-ulation" has much wisdom about future-proofing the data structure.
> The IT industry has huge problem with complexity. We need to simplify the whole computing stack in interest of reliability, security and future innovations.
Yes! I really hope I keep hearing more of this sentiment and that eventually we collectively take action. What would be the first practical step? There's a lot of effort duplicating the same functionality across different languages and frameworks. Is reducing this duplication a good first goal? Should we start at the bottom and convince ARM/x86/AMD64 to use the same instruction set? After that, should we reduce the number of programming languages? It seems there's still a lot of innovation going on, would it be worth stifling that?
The actual non-snarky first step would be to admit that we are over our depth and we can no longer deliver software that is reliable, secure and maintainable. We can only guarantee that our software works for at least some users, on current versions of OS/browser, and is hopefully secure against some of poorer attackers.
Countless variants of programming languages and of instruction sets are not an issue. The problem is lack of well-defined non-leaky interface on boundaries of abstraction layers.
This is too big a topic to reliably cover in a comment (or ten) but standardising using strongly and strictly typed data formats like ASN.1 and EDN and practically forfeiting everything else (JSON, YAML, TOML, INI, XML) for configuration might be a good first step.
You cannot innovate if you keep insisting on eternal backwards compatibility. That's just the facts of life. At some point a backwards compatibility breaking move must be made. It's absolutely unavoidable and we'll see such moves in the near future.
> Is reducing this duplication a good first goal? Should we start at the bottom and convince ARM/x86/AMD64 to use the same instruction set?
Not sure about the CPU architectures; it seems they have been stuck in a local maxima for decades and just in the last few years people started finally asking if there are better ways to do things.
But as for some of the author's points, you can bake in certain services directly in the OS (say, utilise SQL for accessing "files" and "directories" instead of having a filesystem), standardise that and then just make sure you have a good FFI (native interface) to those OS services no matter the programming language you use -- akind to how everybody is able to offload to C libraries, you know?
> After that, should we reduce the number of programming languages?
We absolutely should, even if that leads to street riots. We have too many of them. And practically 90% of all popular languages are held together by spit, duct tape and nostalgia -- let's not kid ourselves at least.
It cannot be that damned hard to identify several desirable traits, identify the languages that possess them, combine that with the knowledge of which runtimes / compilers do the best work (benchmarking the resulting machine code is very good first step in that), then finally combine that with desirable runtime properties (like the fault tolerance and transparent parallelism of Erlang's BEAM VM). Yes it sounds complicated. And yes it's worth it.
> It seems there's still a lot of innovation going on, would it be worth stifling that?
Yes. Not all innovation should see production usage. I can think of at least 10 languages right now that should have remained hobby projects but became huge commercial hits due to misguided notions like "easy to use". And nowadays we no longer want easy to use -- we want guarantees after the program compiles, not being able to spit out a half-working code in 10 minutes (I definitely can't talk about all IT here, of course, but this is a sentiment / trend that seems to get stronger and stronger with time).
Many languages and frameworks aren't much better than weekend garage tinkering projects and should have stayed that way -- Javascript is the prime example.
Most operating systems ship a general-purpose structured binary serialization format parser as an OS component: ASN1. There have over the years been a number of security critical bugs in there, and everybody hates ASN1 anyway.
ASN.1 has an amazing idea and an awful implementation. :(
I'd say standardise a subset of ASN.1's binary and text representations and introduce a completely different schema syntax -- LISP seems like a sane choice -- and just stop there.
ASN.1 suffers the same problems that many other technologies suffer: they have way too many things accumulated on top of one another. Somebody has to put their foot down and say: "NO! Floating-point numbers in these binary streams can be 32 bit, 64 bit and arbitrary precision but no more than 1024 bits! I don't care what you need, there's the rest of the world to consider, deal with it". And people will find a way (maybe introduce a composite type that has 2x 1024-bit floats).
We need standard committees with a bit more courage and less corporate influence.
Well, I don't want to say anything about the utility, longevity or appeal of yaml/json, but I somehow think a user is going to upgrade their entire operating system before they upgrade my little app.
And if they're inclined to upgrade my app, I mean, nothing stops me from using a third party library to parse yaml. It sounds like we're talking about an app from three operating systems and 20 years ago so it's likely I'm doing that anyway - maybe not in the current Windows version, but in a recent enough version on some other operating system.
> Given a program supporting 16 bit address spaces from the 1970s, you can load it into a modern x86 OS today and it works.
Actually, it doesn't. It is extremely hard to properly return to 16-bit userspace code from a 64-bit kernel, so Windows removed support for it entirely, and it's not enabled by default on Linux.
Old operating systems like VMS, MULTICS, OS/400, etc. aimed for this; they were large and contained support for a lot of structure. Problem is, they always evolved to be too complex, and/or the available complexity was not what turned out to be needed, but different complexities had to be built on top of the old, unneeded, complexities.
Along comes Unix and dispenses with all of that. Files? Bytes only. Devices? Accessible as files in the file system. Etc. This became popular for a reason.
I came here to write this. VM/CMS was designed for processing structured data. Application UIs were designed around forms to be displayed on 3270 terminals and data was structured around what could be input on a single punch card. It was great as long as this fit your model.
What UNIX gave the world was maximum flexibility: an os that only really cared about streams and got out of your way.
And in my eyes this is the only right way, because it allows to build structured services on top. If on the other hand the structure is already in the underlying system, it's incredibly hard to build something useful on top.
Similarly, think how useful memcpy() is: Because it can be applied anywhere.
The important question here is, why? Was it because of those features? Or something else?
I don't know, but one major factor in OS adoption is network effects. "The rich get richer", and it's very hard to introduce a new and different system, no matter how good it is.
Unlike Unix, which was basically free software and therefore extendable and available for companies to build their own versions of, Plan 9 was proprietary from Bell Labs.
Having cruft in your top layer is fine and healthy; a sign of a living, evolving system that is still learning how best to do its job and hasn’t prematurely ossified.
The problems occur when a still-crufty layer becomes set in stone, ensuring its base defects go on to pollute everything built on top. That may happen either because its designers lack perspective and cannot accept their precious design is flawed; or because everyone else starts building on top before it is ready for prime-time, then reject all attempts to straighten it out because that would require them to rewrite as well.
Worth bearing in mind here that files and file systems are themselves a kludge, a 1950s workaround for not having enough primary storage to keep all programs and data live at all times.† (Also, primary storage traditionally tends to be volatile; not good when the power goes out.) I point this out because institutionalizing kludges as The Way Things Should Be Done is an awfully easy mistake to make, and in a long-lived systems like OSes have serious architectural sequelae.
..
What’s interesting about the Unix File System is that it’s a general abstraction: a hierarchical namespace that can be mapped to a wide range of resources; not just “files” in secondary storage but also IO devices, IPC communication points, etc. And that’s all it did: mount resources at locations and define a common API for reading/writing/inspecting all resources, without getting bogged down on the internal details of each resource. Nice high cohesion, low coupling design.
Plan 9 made much fuller use of the namespace idea than Unix did, but the core concept was there for the start and it is excellent… except for one MASSIVE mistake: individual resources are neither typed nor tagged.
Without formal type information there is no way for a client to determine the type of data that a given resource represents. Either there is a common informal agreement that a resource mounted at location X is of type Y (e.g. /dev/*), OR there is an informal resource naming convention (typically DOS-style name extensions), OR the client has to guess (e.g. by sniffing the first few bytes for “tells” or by assuming ASCII).
Formally tagging each resource with, say, a MIME type (as in BeFS, or properly implemented HTTP) would’ve made all the difference. THAT is K&R’s million-dollar mistake mistake, because without that key metadata it is impossible to ensure formal correctness(!) or implement intelligent helpers for data interchange (e.g. automatic coercions).
Arguments over the pros and cons of alternative namespacing/association arrangements (e.g. relational vs hierarchical) are all secondary to that one fundamental cockup.
..
Unix became popular not because it was Good, but because it was Just Good Enough to entertain the cowboys who built and used it, who enjoy that sort of pants-down edge-of-the-seat living. And because a lot of them were in education, they spread it; and so made cowboy coders the status quo for much of this profession and culture. And while I admire and largely approve of their Just Do It attitude, I abhor their frightening disregard for safety and accountability.
And while I’m heartened to see some signs of finally growing up (e.g. Rust), there is a LOT of legacy damage already out there still to be undone. And retrofitting fixes and patches to endemic, flawed systems like Unix and C is and will be infinitely more pain and work than if they’d just been built with a little more thought and care in the first place.
--
† If/When memristors finally get off the ground, the primary vs secondary storage split can go away again and we finally get back to single flat storage space, eliminating lots of of complex bureaucracy and ritual. And when it does, there’ll still the need for some sort of abstract namespace for locating and exchanging data.
> a 1950s workaround for not having enough primary storage to keep all programs and data live at all times
Even today you don't have enough primary storage to keep your data. And even then it would require a structure when data outlives the process / application.
Most times there are no best solution, but only tradeoffs. Anyone who has done a bit of systems work knows this. And Hierarchical file system was a Ok-ish trade-off to make. Perfect is enemy of good.
> MASSIVE mistake: individual resources are neither typed nor tagged.
It comes with its own set of tradeoffs. I am a huge proponent of static typing when it comes to PLs. But in a system where multiple actors operate on shared resources, it is easy to get illusioned into a false sense of correctness. Also it imposes some extra complexity in the programming model. I am no experienced systems engineer. But someone here can address it better.
> .... entertain the cowboys who built and used it ....
You are going beyond HN standards to justify your anger against a particular methodology or people that embrace it in programming.
The universally accepted point is that Unix succeeded due to political factors (low cost and easy modification compared to proprietary counterparts), simplicity of the API, and being arguably better than others despite lacking some features people love to lament these days. But in many cases, that simplicity is a desirable thing to have. It is nice to objectively point out faults in systems. But what you did is totally dismissing some people's contributions.
It is easy to see some hyped thing and think that's the Next Big Thing(TM) after reading two fanboys preaching on Reddit, while being totally ignorant of tradeoffs.
> Unix became popular not because it was Good, but because it was Just Good Enough to entertain the cowboys who built and used it ... and so made cowboy coders the status quo for much of this profession and culture
I disagree very strongly with these insults directed at programmers from 50 years ago because now, in retrospect, half a century later what they did doesn't live up to some flawless system written in Rust that exists only in one's imagination.
Doesn't is seem a little bit like calling Thomas Edison a cowboy that made the terrible mistake of giving us electric lighting through filament bulbs when LED lights would have been so much better.
In these early days of computer science I read virtually every important published article on programming languages and operating systems, the field was still that small. MIT didn't even think it warranted a separate department, it was just a subsidiary branch of EE like say communications. Researchers like Edsger Dijkstra, Tony Hoare, Niklaus Wirth, Per Brinch Hansen, Leslie Lamport, David Gries, Donald Knuth, Barbara Liskov, and David Parnas were all trying to figure out how to structure an operating system, how to verify that a program was correct, how to solve basic problems of distributed systems, and how to design better programming languages. Practitioners working on operating systems would have been familiar with almost everything written by these giants.
It's easy to insult C, I myself wouldn't choose it for work today. But in 1989, 20 years after the birth of Unix, I did choose it for my company's development of system software--it still made sense. And back in the 1960's what alternatives were there? Fortran? PL/1? Pascal? Lisp? We were still programming on keypunch machines and relational databases hadn't been invented. The real competition back then for system programming was assembly language.
> Formally tagging each resource with, say, a MIME type (as in BeFS, or properly implemented HTTP) would’ve made all the difference.
That gives you a "global naming of things" problem, which is surprisingly hard. Who controls the namespace? Who gets to define new identifiers? Do they end up as intellectual property barriers where company A can't write software to work with files of company B?
> without that key metadata it is impossible to ensure formal correctness(!)
That seems irrelevant - even with the metadata you have to allow for the possibility of a malformed payload or simple metadata mismatch. I don't believe this alone would prevent people from sneaking attack payloads through images or PDFs, for example.
> THAT is K&R’s million-dollar mistake mistake
K&R wrote UNIX before MIME. Not only that, but before the internet, JPEG, PDF, and indeed almost all the file types defined in MIME except plain text.
Refusing to choose also prevented UNIX from being locked into choices that later turned out to be inconvenient, like Windows deciding to standardise on UCS-2 too early rather than wait for everyone to converge on UTF-8.
Even the divergent choice of path separators and line endings has turned out to be a mess.
> cowboys
The "cowboy" system is the one that beat the others, many of which never launched (WinFS, competing hypertext protocols) or were commercially invisible (BeOS, Plan9 etc).
Both Windows and MacOS have alternate file streams which can be used for metadata, but very rarely are.
Memristors aren't going to save you either. Physical space ultimately determines response time. You can only fit so much storage inside a small light cone. We're going to end up with five or six different layers at different distances from the CPU, plus two or three more out over the network, getting cheaper and slower like Glacier.
We probably are going to move to something more content-addressable, in the manner of git blobs or IPFS, and probably a lot closer to immutable or write-once semantics because consistency is such a pain otherwise. It would be interesting to see a device offering S3-style blob interface plus content-addressable search ... over the PCIe interface.
Oh, and there's a whole other paper to be written on how access control has evolved from "how can we protect users from each other, but all the software on the system is trusted" to "everything is single-user now, but we need to protect applications from each other".
> That gives you a "global naming of things" problem, which is surprisingly hard. Who controls the namespace? Who gets to define new identifiers? Do they end up as intellectual property barriers where company A can't write software to work with files of company B?
Well, you could go the sqlite route: if you want the DB to be globally accessible for everyone else, pass an absolute path; if you want it to be anonymous and for your temporary usage only, pass ":memory:" as the DB path.
Additionally, UUIDs / ULIDs / what-have-you can be extended but IMO just make 512-bit global identifiers the norm and say "screw it" for the next 1_000 - 100_000 years. (While making a standard library to work with those IDs baked in every popular OS's kernel).
Sure, it might get complicated. Still, let it be one complicated thing on one specific topic. Still a better future status quo compared to what we have today.
---
I mean, nobody here seems to claim that changing things will be instantly easier and stuff will automagically fall in place in a year. No reasonable person claims that.
But freeing us from the "everything is strings on shell" and the "zero conventions" conundrums themselves will probably solve like 50% of all technical problems in programming and system/network administration.
(While we're at it, we should redefine what the hell a "shell" even is.)
One thing that annoys me is that when "directing" other programs, you need to write tests that check if each program is alive and kicking. If you for example is checking a http server, you need a http client to test it. And the most frustrating part is that you don't know when the program is ready to do whatever it's supposed to do, so you have to constantly poll until it answers, but then when you disconnect it might have some warmup/warmdown mechanism, so you can't put it to use right away, you have to do manually testing - then set a timeout with a good margin. Then when you "director" worker crashes (and restarted by another "init" service) those children are still alive, so when you try to start another instance it will complain about the port being used, so the "director" worker also have to check for existing orphans.
.. It's sometimes easier to just rewrite the program yourself so it becomes a port of you "monolith".
I like the unix philosophy and micro-services. But I feel it can be better...
Imagine interoperability nightmare when we cannot rely on everything being just bytes being streamed.
I mean, everything would be stored and transmitted one time or another. Word document, sqlite database, an email and it's attachment. Imagine you could not send something as simple as word document because ip protocol assumes stream of bytes, and your operating system talks custom storage format. Imagine you cannot store, use sqlite database efficiently because operating system does not present you with efficient, fast, compatible byte storage.
“ Imagine interoperability nightmare when we cannot rely on everything being just bytes being streamed.”
As I’ve noted above, the problem isn’t transmittability; the problem is never knowing what the bytes being transmitted represent.
I mean, C is hardly renowned for the robustness or expressivity of its “type” system, but untagged untyped byte streams are tantamount to declaring all data as void*. That is a ridiculously shaky foundation to build on, yet could have been entirely avoided by simple addition of one more piece of metadata and an ftype() API.
K&R were brilliant, but also kinda dumb. I certainly wouldn’t want to eat chicken cooked by either one.
I think the author was more advocating for having something like sqlite being a part of the kernel and the de facto way of accessing a lot of data (as opposed to files and directories and byte streams in general).
There are a bunch of things that don't really make sense in modern software. For example, we now commonly package things up as docker containers with immutable/ephemeral file systems. For better or for worse, that's how a lot of server-side stuff gets deployed.
The whole point of that is simply emulating what the program is expecting to such an extent that it can run as if it was running on a normal file system. Often that means working around many broken assumptions. For example it might expect some configuration in a particular place in the filesystem. However, since the filesystem is immutable, you can't really modify that after you build the docker container. So you work around it by e.g. using dockerize to template the config file and then inject the actual configuration from the environment variables on the docker command line. Likewise, applications produce logs but writing those to an ephemeral file system is kind of pointless; so you instead write to stderr/stdout and leave it to the host OS, Docker, Kubernetes, or whatever is running your container.
These workarounds are something to avoid if you are writing new software. When you ship as a Docker container, you don't care about the host OS. It might be Linux, BSD, Mac, Windows, or whatever else is capable of running docker containers these days. Chances are that the host OS itself is also running in some VM if you are shipping to a cloud environment. The nice thing with Docker is that you (mostly) don't have to care about any of that.
With WASM this is extending to other places. People are running edge computing functions in WASM form, running WASM programs in a browser, or even in operating system kernels. Most of those places don't necessarily even have filesystems, environment variables, or log files that you can access (or should access).
IMHO modern development requires a bit of upfront thinking on how to configure things, where to send logs, and where to store data. Rarely is the answer for any of those things a file on an (ephemeral) disk. Logs need to be aggregated so they can be analyzed. Local storage is not always available/reliable. Configuration gets injected rather than loaded from a file. Files/state gets written to some specialized service (a networked file bucket, some DB, a queue, REST service, etc.). Most of those things are accessed via networking rather than a file system.
This is also true of most frontend development these days. A browser application does not have a filesystem; it has no access to environment variables; and while it has a console, it's considered bad form to actually log to it in production apps. Instead browser apps write to remote services (including logging and analytics data) and get their configuration from things like cookies, in browser databases, and remotely stored user preferences.
Come to think of it, most modern development is kind of detached from the operating system these days.
Ehhh... Why is that the obvious solution? We can't decide on the right way to do it in user space, why does moving the problem to the OS help? This seems to be based on the whimsical idea that having the OS do it would somehow fix the varied problems of structured communication. Are we enforcing WSLDs in the OS? One size fits all structures defined by the OS? I don't think the rambling thoughts really made it back to the thesis.
That said, I suggest to anyone interested in this stuff to try Powershell...no really! I don't use it often but it is a window into another world where everything has a structured definition behind the text output.
> Problem: We make a lot of CRUD apps.
> Solution: The OS should do it.
>
> Ehhh... Why is that the obvious solution? We can't decide on the right way to do it in user space, why does moving the problem to the OS help?
The article is indeed ridiculous. An OS should not do everything. Hardware storage resources are generally the memory, disk, and network connection (and if you're getting really deep, the cache and registers). A good OS should only provide access to those resources as efficiently as possible across a wide variety of hardware.
There is a vast myriad of ways of utilizing those resources, and it would be a fools errand to implement a one-size-fits-all approach. The better approach is to provide access to the resources, and let higher level software developers build on top of them.
A disk only database is far different than a disk database with a memory cache, is far different than a memory only database, is far different than any collection of the above coordinated via a network connection. Further, storing text is different than storing images, which is different from storing video, which is different from storing JSON or XML.
Pushing everything to the OS will often give you worse performance, locks you into a single OS vendor, and slows down innovation from third parties. Bad idea.
> An OS should not do everything. Hardware storage resources are generally the memory, disk, and network connection (and if you're getting really deep, the cache and registers). A good OS should only provide access to those resources as efficiently as possible across a wide variety of hardware.
It sounds like you're talking about kernel only. So I guess your OS of choice is something like LFS?
My view of an operating system is very different; it's supposed to be a complete system ready for productive work as well as a programming environment and platform for additional third party software.
> Pushing everything to the OS will often give you worse performance, locks you into a single OS vendor, and slows down innovation from third parties. Bad idea.
Pushing everything to third parties will often give you massive duplication of effort and dependencies, excessively deep stacks that eat performance and make debugging harder, locks you into a clusterfuck of dependency hell, and slows innovation from first party because now they must be very sure not to break the huge stack of third party stuff that everyone critically depends on. There'll be no cohesion because third parties invent their own ways of doing things as the stripped-to-the-bones OS has no unified vision, documentation is going to be all over the place, there's nothing holding back churn... development of third party applications is slow and frustrating because the lowest common denominator (underlying OS) is basically magnetic needle. Bad idea.
This is largely why I prefer BSD over Linux, but I share the author's frustrations with Unix in general.
I always disliked the file systems being hierarchical databases with the limitations they have.
Even from the simple, non programmer user (or a "power user") perspective I don't actually want the files the way they are. A file usually needs a name (which is not good for anything) and stores both the data and the metadata inside, a metadata correction automatically changes the file hash and the file system doesn't consider the fact most of my files (or at least the data part of them) are never meant to change (taking this fact in account could provide extra optimization and protection). Ideally a file should only store data (e.g. a pure sound stream for an audio file) + a collection of metadata fields (including tags!) I can query to find the files I need. AFAIK something like that was supposed to be implemented in WinFS but it was cancelled.
In fact you can use extended attributes/streams or even use SQLite as a file system (or a file format). But nobody does that. I feel interested in implementing a file manager to store all my files in SQLite and provide a convenient GUI to work with them but it seems too much work (I don't even have an idea of the UI) and its usefulness is going to be questionable as no application developed by others is going to be able to open files directly from that database anyway. Nevertheless it can be a good idea to use SQLite as a file system for particular projects like those you have described.
> its usefulness is going to be questionable as no application developed by others is going to be able to open files directly from that database anyway
Don't most operating systems provide for some kind of vfs nowadays? So you can indeed expose it as a legacy hierarchical filesystem if you want.
I guess the reason every extended database type filesystem has failed is the same one: basically, you need legacy support. Otherwise the user agents don't exist to access your data. Several older operating systems had support for attributes of various type and purpose, but they roughly died as the web became popular, because HTTP has no way to handle that problem.
The nearest exception is on handheld devices that run non-legacy operating systems, but then power users complain they don't have adequate control.
> but then power users complain they don't have adequate control.
They wouldn't if they were actually given adequate control. The problem of iOS is not the fact it doesn't expose a traditional file system, it's the fact it doesn't expose any and won't let you control the apps beyond the level Apple likes nor to sideload custom apps you or the free software community might have developed (which certainly isn't a 100% evil policy - it actually protects you from evil agents like the Chinese TSA which is known to install hidden apps on Android phones it searches). I couldn't even find a way to play OPUS audio files on a recent iPhone (despite the fact iPhone supports OPUS in hardware). As a power user I can be pretty well-satisfied without access to the file system if only the UIs and APIs above it provided the level of control and functionality I want.
Well, you convinced me. It's an interesting project idea.
Concerning user interface, I think the ordinary user interface of such a system would have to be application based. Rather than increasingly impure desktop metaphor, you simply list the applications available to user. They open the application and select from the available documents. You would have some kind of list of filters, including automatically maintained filters (like "time last opened") and manually maintained tags.
You could probably have a power users interface that is "the file manager". But it's just the normal file picker, without a default filter of "files i know how to open".
But you still have a problem when you export a file. How do you export that compressed audio stream when you want to send it to your friend? You need to identify all the metadata and reconstruct an MP3 file. Presumably you could implement version 0 of this without changing the file storage, since file importer/exporter is kinda crucial. So you just cache the important data somewhere.
It would all be more important if files were still as relevant as they used to be. I used to have ogg vorbis files and photos and everything. Nowadays, it's really only code and occasionally a document. Otherwise I'm in my web browser or on an app on my phone.
What I can conclude, however, is the idea we shouldn't choose one. Traditional FS should be kept there (for now at least, mostly because it's going to be an extremely hard to replace it with anything which would make more sense for the task) to store system files (and code files perhaps) + a better storage for user files (like pictures, videos, audios, documents etc) and this is mostly a matter of inventing a proper file manager. As for the projects where you are to store enormous numbers of files (so inode number limitation becomes a problem) like mentioned in the article there is little point in using a general purpose FS in the first place, one should either use a database or a raw hard drive (or a database on a raw hard drive).
If one application is able to use it or has a version able to use it the users of that application could format a drive for it. More applications could follow if the gains are big enough. Hardware could be created to fit better. Again, if the gains are big enough it is worth it. More applications will follow. Different things could migrate there like those SaaS like functions. Data could be shared between applications.
Inserting data A into B, serializing it into C, storing it as D then morph it back into C, turn it back into B again so that you can read A from it???
If A is 10 bytes, B is 10 GB OR we don't know if A exists in C? The ease of use and/or performance gain would be substantial even without memorizing 1000 manuals worth of exotic formats.
194 comments
[ 2.2 ms ] story [ 225 ms ] threadIt's like critiquing a master craftman's tools by observing how a master craftman uses his tools, and then critiquing how the apprentice uses those tools. Sure, some of that criticism of the concepts is valid, but they picked the worst implementation of those concepts to critique.
One of the things mentioned in the critique is how the filesystem in question communicates the changes back to the user, and how the user is expected to know what parts of the filesystem to alter. While in the examples given, it does seem relatively obscure, the decry the fact that they had to check the manual for the position to write to.
However, both of those would still be the case for the system that they talk about. Let's talk about object orientated systems, which already exist, in limited (compared to the operating system) forms. You still have to pick up a manual to figure out what keys and functions are relevant to you. In Pharo (which is more or less the system they're after!), you can inspect objects that exist and the methods that exist for them -- however -- often, similarily-named functions do different things, which also requires reading a manual. In addition, using these tools as a first-time user, I was overwhelmed by the number of functions available, most of which I could only guess at what their purpose was. Pharo integrates the manual into the system, but the manual is still there!
Standardized error codes are given in structures passed back (As in Erlang) or in Exceptions, but there are problems with those too. You still have to figure out what those exceptions or errors mean, and there isn't necessarily a standardized format for those errors. As a developer I've been recently been working on writing a Python api. As someone who has little experience with the intimate details of the Python language, I do not always know of the exceptions that exist that I am able to throw, I do not always know of which exceptions would be most appropriate. The same problem exists (obscure error values), but in a different form. There are existing libraries that I rely on that implement more exceptions, sometimes these aren't documented, and even with the venerable Requests library, I have still had to crack open the objects that exist to find easier methods of passing data that the Requests library has, and uses, but does not document for external use.
Let's look at the windows hive database. That's a database of keys for values in the operating system. As a random developer, would you be able to open it and figure out what it does? I wouldn't. As a windows power user I often relyied on articles from howtogeek without really fully understanding what the keys were doing (Although because of my experience an a systems developer, I could guess, but only after the fact). Again, the same problem ("I know what I want to do, but I do not know how to do it") is exposed in a different form, and the methods and practices of the Microsoft organization make that difficult hard to reach. Yet again, the same problems are there, but in a different form.
I do agree that the shell should handle program arguments, a program could expose keys and values and a shell could read that and tell the user about it. I would be interested to see the problems that the Oil shell encounter in their implementation of an object-orientated shell.
Oberon is on my todolist, Inferon is new for me.
> They say Plan9 is a step in the right direction, but rather than attacking a specific aspect of Plan9, they critique how the concepts were imported into Linux, which seems inherently silly to me.
Well, I've played with Plan9 several times, but I don't really feel like criticizing it, because I don't really know it that much. Only thing worth of criticizing is that it almost feels like objects, but not implemented fully.
I agree, that criticism of Plan9 would make more sense from philosophical point of view, but it wouldn't be authentic.
> However, both of those would still be the case for the system that they talk about. Let's talk about object orientated systems, which already exist, in limited (compared to the operating system) forms. You still have to pick up a manual to figure out what keys and functions are relevant to you. In Pharo (which is more or less the system they're after!), you can inspect objects that exist and the methods that exist for them -- however -- often, similarily-named functions do different things, which also requires reading a manual. In addition, using these tools as a first-time user, I was overwhelmed by the number of functions available, most of which I could only guess at what their purpose was. Pharo integrates the manual into the system, but the manual is still there!
Thats true, but you can also do a type-checking and infer a lot only from the proper naming of the objects and methods in the "Clean code" (book) style.
> Let's look at the windows hive database. That's a database of keys for values in the operating system. As a random developer, would you be able to open it and figure out what it does? I wouldn't.
One of the things that is not strongly captured in this article and which I since consider increasingly important is ability to use reflection on the system. Database that you can't open and figure out what it does is not worth using.
I wish sites wouldn't try annoying tricks like this.
When you start dealing with 500,000 files in one directory and you try to do a files->getname(10)->size->to_string(). You are bound to run out of physical memory. This is probably why newer operating systems are so slow. Caching will only save you for so long.
Also your example is weird and you won't run out of memory if you use generators.
General purpose editing of structured data. Org-mode is the closest thing I know of but it is still infuriatingly ad-hoc and text-first.
Grepping (search in) and diff/patch of structured data. While many text manipulations are easy to implement in search/replace manner, structured data are more involved.
Errors and other kinds of unexpected input/output. Semantics of unix process failure are clear. State of the system after one process crashes is readily known. We can compare the filesystem before/after.
If we instead throw an exception in object-oriented runtime, the result is not so easy to analyze or compare, there may be dangling references or other complications. I don't know of any language runtime that can dump itself whole in (semi-)readable format for later comparisons.
One of the undersold features of plain text is being able to exist in "invalid" states while it's being edited. Structured data UI needs to have a least that level of malleability to have a chance over the current paradigm.
Comments should actually be independent of the underlying structure they're documenting. I'd like them to be an independent layer of annotations that can be applied to any data.
Structured data passing between programs instead of just text is part of Powershell concept.
Calling of other programs to request specific actions with smooth UI called Android intents.
If you want to store structured data, you should use well, a database.
So, part of critique of author is Linux specific.
But generally I agree with author: OS are poor abstractions and really need to be improved.
dbus, CORBA and COM would like to have couple words with you
i want to note that Inferno, the other OS that came from Bell-Labs in the 90s did have a typed shell called alphabet:
http://www.vitanuova.com/inferno/man/1/sh-alphabet.html
mayhaps you'll find this interesting.
Many a novice programmer has no doubt made the mistake of thinking that you could, for instance, do this to deliver a C/C++ struct across a socket:
that works until the reader and writer are on machines with different architectures.Same observation could be made about filesystems used to store "structured data". We serialize and deserialize it because the in-memory representation is not inherently portable.
Since there's no standard API (blessed by the OS) and the lowest common denominator is byte streams, we're seeing all these ad-hoc solutions and a hodgepodge of formats and libraries to deal with. That's lots and lots of time and money spent on rather basic stuff.
The approach you're describing only works for POD-style "structured data". Once you start using OOP of almost any type (though not every type), you no longer have ... well, POD that you can move to/from a storage medium. You have objects whose in-memory format IS important and compiler dependent.
There are other concerns too. His WAV example (I write pro-audio software for a living) doesn't even begin to touch on the actual complexities of dealing with substantial quantities of audio (or any other data that cannot reasonably be expected to fit into memory). Nor on the cost of changing the data ... does the entire data set need to be rewritten, or just part of it? How would you know which sections matter? Oh wait, the data fundamentally IS a byte stream, so now you have to treat it like that. If you don't care about performance (or storage overhead, but that's less and less of a concern these days), there are all kinds of ways of hiding these sorts of details. But the moment that performance starts to matter, you need to get back the bytestream level.
And so yes, there's no standard API and yes the lowest common denominator is byte streams ... because the __only__ common denominator is byte streams. Thinking about this any other way is a repeat of a somewhat typical geek dream that the world (or a representation of the world or part of it) can be completely ordered and part of a universal system.
Having that high-level knowledge of data structure enables all sorts of intelligent automation.
In the event that the client uses a different memory layout, it could look up a coercion handler that converts the supplied data from its original layout to the layout required by the client. This is, for instance, how the Apple Event Manager was designed to work: all data is tagged with a type descriptor:
typeSInt16, typeSInt32, typeSInt64 typeUInt16, typeUInt32, typeUInt64
typeIEEE32BitFloatingPoint, typeIEEE64BitFloatingPoint, typeIEEE128BitFloatingPoint
typeUTF8Text, typeUnicodeText (UTF16BE), typeUTF16ExternalRepresentation (UTF16 w. endian mark)
typeAEList (ordered collection) typeAERecord (keyed collection)
and so on. (The tags themselves are encoded as UInt32; nothing so advanced as MIME types, but at least they’re compact.)
The AEM includes a number of standard coercion handlers for converting data between different representations, and clients may also supply their own handlers if needed. Thus the server just packs data in its current representation, and if the client uses the same representation then, great, it can use it as-is. Otherwise the client-side AEM automatically coerces the data to the form the client as part of the unpacking process.
There are limits in the AEM’s design, not least the inability to describe complex data (arrays and structs) with a single “generic” descriptor, e.g. `AEList(SInt32)`. That would vastly simplify packing and unpacking—in best case to simple flat serialization/deserialization, at worst to a single recursive deserialization—instead of two recursive operations with lots of extra mallocs and frees for interim data. But the basic principle is sound, and adheres well to the “be liberal in what you accept” principle.
I believe Powershell does something similar when connecting outputs to inputs of different (though broadly compatible) types, intelligently coercing the output data to the exact type the input requires. No manual work required; it “Just Works”.
Or, if you don’t mind the extra overhead then content negotiation is also an option, which is something HTTP does very well (though web programmers very badly). That is advantageous when communicating with “less intelligent” clients as it permits the server, which best understands its own data, to pre-convert (e.g. via lossy coercion) its data to a form the client will accept.
Lots of ways that Unix’s “throw its hands up and dump the problem all over the users” non-answer can be massively improved on, in other words, without ever losing the lovely loose coupling that is a Unix system’s strength. It only requires a single piece of essential—yet missing—information: the data’s type.
Typed data streams were not invented by Apple. Back in the 1980s, there was (for example) Sun's RPC mechanism, which gave you "seamless" remote procedure call, including transfer of arbitrary structures over a network.
But the original post is much more about filesystems. I used the socket example merely to illustrate the problem, not the actual topic.
I started this project several years ago to support live code updates in production systems without sacrificing performance, but I eventually hit the same issues with exchanging structured data: https://github.com/Morgan-Stanley/hobbes
I split out structured logging into a single header-only lib, structured RPC into a single header-only lib, etc. It's not quite the same as having native support in the OS, but because the mechanism is so lightweight (don't even need to link anything special), it's a pretty good substitute.
I wonder if we're past the point of return, though, in terms of technical divergence. It sounds like, in the Ancient Times, there was a handful of great programmers whose work created the world we program in now. But now, there's way more programmers being paid to make slightly different versions of this "next step", and it would require widespread agreement/coordination to implement it on a scale where it's a seamless feature that's taken for granted the way the shell/network/fs are.
No, it is not a so big problem. And no it will not do our life easier. Also author did not mention about the real problem of semantic. How client should interpret a structure to compose a valid request.
OS should not know about userspace structures because OS don't do anything with it. It stores and transfers chunks of bytes and its semantic is defined by userspace. And forcing current popular serializing format on OS level is the most dumb idea ever.
The idea is that you take the common elements of all of those serialization formats and when you take a good look you notice that the lowest common denominator isn't actually raw bytes on a disk.
Seriously though, the question of "why isn't it done this way" is answered a couple times: "it's all just bytes in the end". Bytes & arrays-of-bytes are really the only thing that you can trust that all systems and all languages interpret the same (even 2-byte integers can be interpreted differently by different systems). Presumably the author just wants to be able to memcpy structures between programs or from disk, or maybe with some high efficiency pre-allocated heap stuff, but you really need to have code to validate the bytes, essentially requiring the parsing of your data-structure which the author is trying to avoid.
Running programs with some common argument notation does sound really nice, but ideally that's just calling a function from your language's binding to their library (as their executable should do, right?).
They are not. They are sending a HTTP message, which consists of just octets and has to be parsed. The parsers have to have a ridiculous complexity for various reasons. The HTTP header contains sublanguages such as RFC 7231 §5.3.1 or RFC 8288 §3.
JSON meanwhile has amassed at least six incompatible specifications with the same exact media type, so there is not even a theoretic hope of parsing all messages correctly. https://news.ycombinator.com/item?id=20736665
It links to several RFCs. RFC-7158 and RFC-7159 differ only in (a) that one is RFC-7158 and the other is RFC-7159 (b) that one is dated in March 2013 and the other is March 2014 (c) that one obsoletes RFC-7158 in addition to RFC-4627 which they both obsolete. It appears that 7159 was pushed only to handle an error in the dating of 7158, given that the irrelevant RFC-7157 is dated March 2014.
So to say that there's more that five incompatible specs is just a cruel disregard of the facts. No implementation can differ in its parsing of a text based on the metadata of the spec. There needs to be a substantive difference in the spec.
As to the current RFC 8259, it makes no substantial differences from the previous version. There are a few typos that have been fixed ("though" becoming "through", when "though" was not a possible interpretation). It eliminates the obligation to parse public texts not coded in UTF-8. Previously, documents SHALL be UTF-8 - that's barely a change, but it is a change. It has specified more concretely its relationship with ECMA-404. It's possible that someone reading earlier RFCs would have concluded "a valid RFC 7159 json parser can parse and only the documents produced by ECMA-404". If there were any discrepancies between ECMA-404 and RFC-7159, therefore, an interpretation along these lines would lead a person to conclude you cannot parse a document according to RFC-7159, so the spec doesn't really count as an incompatible spec. Under the current spec, it identifies that there are possible discrepancies, and RFC-8259 is intended to accept fewer documents that ECMA-404.
The first RFC did not permit the json documents "true", "false" and "null", accepting only json documents that comprise an object or an array. It permitted any Unicode encoding, defaulting to UTF-8, and non-Unicode encodings as well (so a Latin-1 document is valid RFC-4627, but not valid RFC-7158+). It also included some incorrect statements about its relationship to javascript. These were reduced in later versions and eventually eliminated. They do not affect the specification of the language, but merely how you may handle data in the language.
No document that is accepted by the current RFC will be rejected by any RFC other than the first. The only such documents to be rejected by the first are those which consist entirely of "true", "false", or "null". No UTF-8 document that is accepted by the first RFC will be rejected by subsequent RFC parsers, unless a person reads a paragraph called "security considerations" that says "you can kinda do this but it's insecure" as somehow trumping the clear statements of the grammar in earlier sections.
I have not investigated the other specs and I probably won't. But the idea that there are 6 incompatible specs is false. There is not more than 5, and it is almost trivial to accept documents according to 3 of 5 or 4 of 6 specs at once by ignoring the restrictions on character coding.
"they only differ in", yeah, indeed.
Dude. We all have work to do and grappling with some obscure JSON parsing corner is the last thing we need in our workday. This applies to 95% of commercial programmers, I am willing to bet.
What you described does sound simple and small in isolation. Now multiply it by 50 and see how "they only differ in" is a problem that must not be allowed in the first place.
Where's the-one-and-only JSON spec? Why no organisation has the courage to step forward and standardise it once and for all?
Given the current realities, just cut your very small and ignorable losses by losing those obscure JSON parsing corners and let people describe JSON simply as:
"RFC #ABCD".
That's it. Nothing else. Only that. Everything is contained in it.
Complexity compounds and makes everything non-deterministic. We should stop allowing that.
I was actually describing something more close to the Smalltalk (http://pharo.org) / Self (http://blog.rfox.eu/en/Series_about_Self.html) image.
Solution: Create a new, unified, universal way of serializing data.
Problem: There are N+1 different ways of serializing data and it's a lot of work to support them all.
Problem: the computer systems we use are insufficiently flexible/too flexible, encouraging the use of N different ways of serializing data.
Solution: make a computer system which encourages centralizing the mechanism by which data is serialized.
In practice, though, you end up with something like XML, which (to steal a phrase, I wish I could remember where I read it) is less a common language than it is an alphabet. I guess it's better than CSV.
For nearly every problem encountered here there's a standard people just ignore. You can implement structured logging at OS level, like Windows does, and see random binary data drops and text files appear all over your logging interface and your file system. Linux config should follow the XDG standard so that configuration for applications can be structured into directories except half the programs don't do that. Windows provides an API for this stuff (the registry) and it's near impossible to configure an application using the registry. The registry is also combined with random config files all over the system, of course, to make it extra difficult to modify a program's behaviour.
Most nice modern features are also missing because operating systems are still being made in C, not C++. The OS doesn't have a concept of objects so yes your socket function will have to fall back to calling select(). A lot of these abstractions for basic communications have been solved on Linux using DBUS, which provides a somewhat standardised interface for many OS daemon and GUI features, all not being used because programmers forget that it exists or because programmers want to use their own solution instead.
On Windows there's COM to try and help with that and well, see where that ended up: a versioned mess of pointers and factories to try to make it easier for everyone, where functionality sometimes completely breaks or needs to be emulated because it turned out the high level concept had a design flaw and now programs won't work if you fix it.
I've done some thought experiments about structuring an OS and a file system to store data consistently and easily parsable, with modern bindings for most concepts. In the end I've had to conclude that the only way to keep the system working like intended would be to either convince everyone to do exactly as I say or to only allow me to write software for such a system. Whatever structure I can think off will inevitably be too constrained for someone else and the middleware abstraction problem starts all over again.
My only conclusion is that I want the OS to be as simple as possible with people following common standards when they write applications, such as using YAML/ini/XML configuration with Syslogd logging and XDG directory structures for user data, with the technical abstractions left at the library level. If we can just get that, most of the inconsistency problem would be solved, but even this is too difficult to do in practice as it turns out.
The system outlined here seems to purposefully avoid it! Some sort of ACID compliant database analogy to a filesystem sounds nice until 20 years down the line when ACIDXYZABC2.4 is released and you have to bend over backwards to remain compatible. Or until Windows has a bug in their OS-native YAML parser (as suggested here) so now your program doesn’t work on Windows until they patch it. But when they do, oh no you can’t just tell your users to download a new binary. Now they have to upgrade their whole OS! Absolute chaos. And if you’re betting on the longevity of YAML/JSON over binary data, well just look at XML.
The proposed improvement as I understood it would be future proof. It seems trivial to build a rock-solid YAML/XML/JSON/EDN parser on OS level, and since it would be so crucial part of OS the mistakes would be caught and fixed quickly. It shouldn't even matter if structured data syntax is replaced or expanded in future, as long as it is versioned and never redacted. Rich Hickey's talk "Spec-ulation" has much wisdom about future-proofing the data structure.
That is literally what Microsoft does.
Yes! I really hope I keep hearing more of this sentiment and that eventually we collectively take action. What would be the first practical step? There's a lot of effort duplicating the same functionality across different languages and frameworks. Is reducing this duplication a good first goal? Should we start at the bottom and convince ARM/x86/AMD64 to use the same instruction set? After that, should we reduce the number of programming languages? It seems there's still a lot of innovation going on, would it be worth stifling that?
Countless variants of programming languages and of instruction sets are not an issue. The problem is lack of well-defined non-leaky interface on boundaries of abstraction layers.
This is too big a topic to reliably cover in a comment (or ten) but standardising using strongly and strictly typed data formats like ASN.1 and EDN and practically forfeiting everything else (JSON, YAML, TOML, INI, XML) for configuration might be a good first step.
You cannot innovate if you keep insisting on eternal backwards compatibility. That's just the facts of life. At some point a backwards compatibility breaking move must be made. It's absolutely unavoidable and we'll see such moves in the near future.
> Is reducing this duplication a good first goal? Should we start at the bottom and convince ARM/x86/AMD64 to use the same instruction set?
Not sure about the CPU architectures; it seems they have been stuck in a local maxima for decades and just in the last few years people started finally asking if there are better ways to do things.
But as for some of the author's points, you can bake in certain services directly in the OS (say, utilise SQL for accessing "files" and "directories" instead of having a filesystem), standardise that and then just make sure you have a good FFI (native interface) to those OS services no matter the programming language you use -- akind to how everybody is able to offload to C libraries, you know?
> After that, should we reduce the number of programming languages?
We absolutely should, even if that leads to street riots. We have too many of them. And practically 90% of all popular languages are held together by spit, duct tape and nostalgia -- let's not kid ourselves at least.
It cannot be that damned hard to identify several desirable traits, identify the languages that possess them, combine that with the knowledge of which runtimes / compilers do the best work (benchmarking the resulting machine code is very good first step in that), then finally combine that with desirable runtime properties (like the fault tolerance and transparent parallelism of Erlang's BEAM VM). Yes it sounds complicated. And yes it's worth it.
> It seems there's still a lot of innovation going on, would it be worth stifling that?
Yes. Not all innovation should see production usage. I can think of at least 10 languages right now that should have remained hobby projects but became huge commercial hits due to misguided notions like "easy to use". And nowadays we no longer want easy to use -- we want guarantees after the program compiles, not being able to spit out a half-working code in 10 minutes (I definitely can't talk about all IT here, of course, but this is a sentiment / trend that seems to get stronger and stronger with time).
Many languages and frameworks aren't much better than weekend garage tinkering projects and should have stayed that way -- Javascript is the prime example.
I'd say standardise a subset of ASN.1's binary and text representations and introduce a completely different schema syntax -- LISP seems like a sane choice -- and just stop there.
ASN.1 suffers the same problems that many other technologies suffer: they have way too many things accumulated on top of one another. Somebody has to put their foot down and say: "NO! Floating-point numbers in these binary streams can be 32 bit, 64 bit and arbitrary precision but no more than 1024 bits! I don't care what you need, there's the rest of the world to consider, deal with it". And people will find a way (maybe introduce a composite type that has 2x 1024-bit floats).
We need standard committees with a bit more courage and less corporate influence.
And if they're inclined to upgrade my app, I mean, nothing stops me from using a third party library to parse yaml. It sounds like we're talking about an app from three operating systems and 20 years ago so it's likely I'm doing that anyway - maybe not in the current Windows version, but in a recent enough version on some other operating system.
Actually, it doesn't. It is extremely hard to properly return to 16-bit userspace code from a 64-bit kernel, so Windows removed support for it entirely, and it's not enabled by default on Linux.
People love to praise "everything is a file", but the other side of the medal is "every piece of configuration is a byte array".
Along comes Unix and dispenses with all of that. Files? Bytes only. Devices? Accessible as files in the file system. Etc. This became popular for a reason.
What UNIX gave the world was maximum flexibility: an os that only really cared about streams and got out of your way.
Similarly, think how useful memcpy() is: Because it can be applied anywhere.
On the other hand, it would take quite a lot of effort to reach a parity of capability for a new OS these days.
[1] https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
I don't know, but one major factor in OS adoption is network effects. "The rich get richer", and it's very hard to introduce a new and different system, no matter how good it is.
(Too bad the replacements aren't always better...)
The problems occur when a still-crufty layer becomes set in stone, ensuring its base defects go on to pollute everything built on top. That may happen either because its designers lack perspective and cannot accept their precious design is flawed; or because everyone else starts building on top before it is ready for prime-time, then reject all attempts to straighten it out because that would require them to rewrite as well.
..
What’s interesting about the Unix File System is that it’s a general abstraction: a hierarchical namespace that can be mapped to a wide range of resources; not just “files” in secondary storage but also IO devices, IPC communication points, etc. And that’s all it did: mount resources at locations and define a common API for reading/writing/inspecting all resources, without getting bogged down on the internal details of each resource. Nice high cohesion, low coupling design.
Plan 9 made much fuller use of the namespace idea than Unix did, but the core concept was there for the start and it is excellent… except for one MASSIVE mistake: individual resources are neither typed nor tagged.
Without formal type information there is no way for a client to determine the type of data that a given resource represents. Either there is a common informal agreement that a resource mounted at location X is of type Y (e.g. /dev/*), OR there is an informal resource naming convention (typically DOS-style name extensions), OR the client has to guess (e.g. by sniffing the first few bytes for “tells” or by assuming ASCII).
Formally tagging each resource with, say, a MIME type (as in BeFS, or properly implemented HTTP) would’ve made all the difference. THAT is K&R’s million-dollar mistake mistake, because without that key metadata it is impossible to ensure formal correctness(!) or implement intelligent helpers for data interchange (e.g. automatic coercions).
Arguments over the pros and cons of alternative namespacing/association arrangements (e.g. relational vs hierarchical) are all secondary to that one fundamental cockup.
..
Unix became popular not because it was Good, but because it was Just Good Enough to entertain the cowboys who built and used it, who enjoy that sort of pants-down edge-of-the-seat living. And because a lot of them were in education, they spread it; and so made cowboy coders the status quo for much of this profession and culture. And while I admire and largely approve of their Just Do It attitude, I abhor their frightening disregard for safety and accountability.
And while I’m heartened to see some signs of finally growing up (e.g. Rust), there is a LOT of legacy damage already out there still to be undone. And retrofitting fixes and patches to endemic, flawed systems like Unix and C is and will be infinitely more pain and work than if they’d just been built with a little more thought and care in the first place.
--
† If/When memristors finally get off the ground, the primary vs secondary storage split can go away again and we finally get back to single flat storage space, eliminating lots of of complex bureaucracy and ritual. And when it does, there’ll still the need for some sort of abstract namespace for locating and exchanging data.
Even today you don't have enough primary storage to keep your data. And even then it would require a structure when data outlives the process / application.
Most times there are no best solution, but only tradeoffs. Anyone who has done a bit of systems work knows this. And Hierarchical file system was a Ok-ish trade-off to make. Perfect is enemy of good.
> MASSIVE mistake: individual resources are neither typed nor tagged.
It comes with its own set of tradeoffs. I am a huge proponent of static typing when it comes to PLs. But in a system where multiple actors operate on shared resources, it is easy to get illusioned into a false sense of correctness. Also it imposes some extra complexity in the programming model. I am no experienced systems engineer. But someone here can address it better.
> .... entertain the cowboys who built and used it ....
You are going beyond HN standards to justify your anger against a particular methodology or people that embrace it in programming.
The universally accepted point is that Unix succeeded due to political factors (low cost and easy modification compared to proprietary counterparts), simplicity of the API, and being arguably better than others despite lacking some features people love to lament these days. But in many cases, that simplicity is a desirable thing to have. It is nice to objectively point out faults in systems. But what you did is totally dismissing some people's contributions.
It is easy to see some hyped thing and think that's the Next Big Thing(TM) after reading two fanboys preaching on Reddit, while being totally ignorant of tradeoffs.
I disagree very strongly with these insults directed at programmers from 50 years ago because now, in retrospect, half a century later what they did doesn't live up to some flawless system written in Rust that exists only in one's imagination.
Doesn't is seem a little bit like calling Thomas Edison a cowboy that made the terrible mistake of giving us electric lighting through filament bulbs when LED lights would have been so much better.
In these early days of computer science I read virtually every important published article on programming languages and operating systems, the field was still that small. MIT didn't even think it warranted a separate department, it was just a subsidiary branch of EE like say communications. Researchers like Edsger Dijkstra, Tony Hoare, Niklaus Wirth, Per Brinch Hansen, Leslie Lamport, David Gries, Donald Knuth, Barbara Liskov, and David Parnas were all trying to figure out how to structure an operating system, how to verify that a program was correct, how to solve basic problems of distributed systems, and how to design better programming languages. Practitioners working on operating systems would have been familiar with almost everything written by these giants.
It's easy to insult C, I myself wouldn't choose it for work today. But in 1989, 20 years after the birth of Unix, I did choose it for my company's development of system software--it still made sense. And back in the 1960's what alternatives were there? Fortran? PL/1? Pascal? Lisp? We were still programming on keypunch machines and relational databases hadn't been invented. The real competition back then for system programming was assembly language.
That gives you a "global naming of things" problem, which is surprisingly hard. Who controls the namespace? Who gets to define new identifiers? Do they end up as intellectual property barriers where company A can't write software to work with files of company B?
> without that key metadata it is impossible to ensure formal correctness(!)
That seems irrelevant - even with the metadata you have to allow for the possibility of a malformed payload or simple metadata mismatch. I don't believe this alone would prevent people from sneaking attack payloads through images or PDFs, for example.
> THAT is K&R’s million-dollar mistake mistake
K&R wrote UNIX before MIME. Not only that, but before the internet, JPEG, PDF, and indeed almost all the file types defined in MIME except plain text.
Refusing to choose also prevented UNIX from being locked into choices that later turned out to be inconvenient, like Windows deciding to standardise on UCS-2 too early rather than wait for everyone to converge on UTF-8.
Even the divergent choice of path separators and line endings has turned out to be a mess.
> cowboys
The "cowboy" system is the one that beat the others, many of which never launched (WinFS, competing hypertext protocols) or were commercially invisible (BeOS, Plan9 etc).
Both Windows and MacOS have alternate file streams which can be used for metadata, but very rarely are.
Memristors aren't going to save you either. Physical space ultimately determines response time. You can only fit so much storage inside a small light cone. We're going to end up with five or six different layers at different distances from the CPU, plus two or three more out over the network, getting cheaper and slower like Glacier.
We probably are going to move to something more content-addressable, in the manner of git blobs or IPFS, and probably a lot closer to immutable or write-once semantics because consistency is such a pain otherwise. It would be interesting to see a device offering S3-style blob interface plus content-addressable search ... over the PCIe interface.
Oh, and there's a whole other paper to be written on how access control has evolved from "how can we protect users from each other, but all the software on the system is trusted" to "everything is single-user now, but we need to protect applications from each other".
Well, you could go the sqlite route: if you want the DB to be globally accessible for everyone else, pass an absolute path; if you want it to be anonymous and for your temporary usage only, pass ":memory:" as the DB path.
Additionally, UUIDs / ULIDs / what-have-you can be extended but IMO just make 512-bit global identifiers the norm and say "screw it" for the next 1_000 - 100_000 years. (While making a standard library to work with those IDs baked in every popular OS's kernel).
Sure, it might get complicated. Still, let it be one complicated thing on one specific topic. Still a better future status quo compared to what we have today.
---
I mean, nobody here seems to claim that changing things will be instantly easier and stuff will automagically fall in place in a year. No reasonable person claims that.
But freeing us from the "everything is strings on shell" and the "zero conventions" conundrums themselves will probably solve like 50% of all technical problems in programming and system/network administration.
(While we're at it, we should redefine what the hell a "shell" even is.)
https://en.wikipedia.org/wiki/WinFS
I mean, everything would be stored and transmitted one time or another. Word document, sqlite database, an email and it's attachment. Imagine you could not send something as simple as word document because ip protocol assumes stream of bytes, and your operating system talks custom storage format. Imagine you cannot store, use sqlite database efficiently because operating system does not present you with efficient, fast, compatible byte storage.
As I’ve noted above, the problem isn’t transmittability; the problem is never knowing what the bytes being transmitted represent.
I mean, C is hardly renowned for the robustness or expressivity of its “type” system, but untagged untyped byte streams are tantamount to declaring all data as void*. That is a ridiculously shaky foundation to build on, yet could have been entirely avoided by simple addition of one more piece of metadata and an ftype() API.
K&R were brilliant, but also kinda dumb. I certainly wouldn’t want to eat chicken cooked by either one.
The whole point of that is simply emulating what the program is expecting to such an extent that it can run as if it was running on a normal file system. Often that means working around many broken assumptions. For example it might expect some configuration in a particular place in the filesystem. However, since the filesystem is immutable, you can't really modify that after you build the docker container. So you work around it by e.g. using dockerize to template the config file and then inject the actual configuration from the environment variables on the docker command line. Likewise, applications produce logs but writing those to an ephemeral file system is kind of pointless; so you instead write to stderr/stdout and leave it to the host OS, Docker, Kubernetes, or whatever is running your container.
These workarounds are something to avoid if you are writing new software. When you ship as a Docker container, you don't care about the host OS. It might be Linux, BSD, Mac, Windows, or whatever else is capable of running docker containers these days. Chances are that the host OS itself is also running in some VM if you are shipping to a cloud environment. The nice thing with Docker is that you (mostly) don't have to care about any of that.
With WASM this is extending to other places. People are running edge computing functions in WASM form, running WASM programs in a browser, or even in operating system kernels. Most of those places don't necessarily even have filesystems, environment variables, or log files that you can access (or should access).
IMHO modern development requires a bit of upfront thinking on how to configure things, where to send logs, and where to store data. Rarely is the answer for any of those things a file on an (ephemeral) disk. Logs need to be aggregated so they can be analyzed. Local storage is not always available/reliable. Configuration gets injected rather than loaded from a file. Files/state gets written to some specialized service (a networked file bucket, some DB, a queue, REST service, etc.). Most of those things are accessed via networking rather than a file system.
This is also true of most frontend development these days. A browser application does not have a filesystem; it has no access to environment variables; and while it has a console, it's considered bad form to actually log to it in production apps. Instead browser apps write to remote services (including logging and analytics data) and get their configuration from things like cookies, in browser databases, and remotely stored user preferences.
Come to think of it, most modern development is kind of detached from the operating system these days.
Solution: The OS should do it.
Ehhh... Why is that the obvious solution? We can't decide on the right way to do it in user space, why does moving the problem to the OS help? This seems to be based on the whimsical idea that having the OS do it would somehow fix the varied problems of structured communication. Are we enforcing WSLDs in the OS? One size fits all structures defined by the OS? I don't think the rambling thoughts really made it back to the thesis.
That said, I suggest to anyone interested in this stuff to try Powershell...no really! I don't use it often but it is a window into another world where everything has a structured definition behind the text output.
The article is indeed ridiculous. An OS should not do everything. Hardware storage resources are generally the memory, disk, and network connection (and if you're getting really deep, the cache and registers). A good OS should only provide access to those resources as efficiently as possible across a wide variety of hardware.
There is a vast myriad of ways of utilizing those resources, and it would be a fools errand to implement a one-size-fits-all approach. The better approach is to provide access to the resources, and let higher level software developers build on top of them.
A disk only database is far different than a disk database with a memory cache, is far different than a memory only database, is far different than any collection of the above coordinated via a network connection. Further, storing text is different than storing images, which is different from storing video, which is different from storing JSON or XML.
Pushing everything to the OS will often give you worse performance, locks you into a single OS vendor, and slows down innovation from third parties. Bad idea.
It sounds like you're talking about kernel only. So I guess your OS of choice is something like LFS?
My view of an operating system is very different; it's supposed to be a complete system ready for productive work as well as a programming environment and platform for additional third party software.
> Pushing everything to the OS will often give you worse performance, locks you into a single OS vendor, and slows down innovation from third parties. Bad idea.
Pushing everything to third parties will often give you massive duplication of effort and dependencies, excessively deep stacks that eat performance and make debugging harder, locks you into a clusterfuck of dependency hell, and slows innovation from first party because now they must be very sure not to break the huge stack of third party stuff that everyone critically depends on. There'll be no cohesion because third parties invent their own ways of doing things as the stripped-to-the-bones OS has no unified vision, documentation is going to be all over the place, there's nothing holding back churn... development of third party applications is slow and frustrating because the lowest common denominator (underlying OS) is basically magnetic needle. Bad idea.
This is largely why I prefer BSD over Linux, but I share the author's frustrations with Unix in general.
Even from the simple, non programmer user (or a "power user") perspective I don't actually want the files the way they are. A file usually needs a name (which is not good for anything) and stores both the data and the metadata inside, a metadata correction automatically changes the file hash and the file system doesn't consider the fact most of my files (or at least the data part of them) are never meant to change (taking this fact in account could provide extra optimization and protection). Ideally a file should only store data (e.g. a pure sound stream for an audio file) + a collection of metadata fields (including tags!) I can query to find the files I need. AFAIK something like that was supposed to be implemented in WinFS but it was cancelled.
In fact you can use extended attributes/streams or even use SQLite as a file system (or a file format). But nobody does that. I feel interested in implementing a file manager to store all my files in SQLite and provide a convenient GUI to work with them but it seems too much work (I don't even have an idea of the UI) and its usefulness is going to be questionable as no application developed by others is going to be able to open files directly from that database anyway. Nevertheless it can be a good idea to use SQLite as a file system for particular projects like those you have described.
Don't most operating systems provide for some kind of vfs nowadays? So you can indeed expose it as a legacy hierarchical filesystem if you want.
I guess the reason every extended database type filesystem has failed is the same one: basically, you need legacy support. Otherwise the user agents don't exist to access your data. Several older operating systems had support for attributes of various type and purpose, but they roughly died as the web became popular, because HTTP has no way to handle that problem.
The nearest exception is on handheld devices that run non-legacy operating systems, but then power users complain they don't have adequate control.
They wouldn't if they were actually given adequate control. The problem of iOS is not the fact it doesn't expose a traditional file system, it's the fact it doesn't expose any and won't let you control the apps beyond the level Apple likes nor to sideload custom apps you or the free software community might have developed (which certainly isn't a 100% evil policy - it actually protects you from evil agents like the Chinese TSA which is known to install hidden apps on Android phones it searches). I couldn't even find a way to play OPUS audio files on a recent iPhone (despite the fact iPhone supports OPUS in hardware). As a power user I can be pretty well-satisfied without access to the file system if only the UIs and APIs above it provided the level of control and functionality I want.
Concerning user interface, I think the ordinary user interface of such a system would have to be application based. Rather than increasingly impure desktop metaphor, you simply list the applications available to user. They open the application and select from the available documents. You would have some kind of list of filters, including automatically maintained filters (like "time last opened") and manually maintained tags.
You could probably have a power users interface that is "the file manager". But it's just the normal file picker, without a default filter of "files i know how to open".
But you still have a problem when you export a file. How do you export that compressed audio stream when you want to send it to your friend? You need to identify all the metadata and reconstruct an MP3 file. Presumably you could implement version 0 of this without changing the file storage, since file importer/exporter is kinda crucial. So you just cache the important data somewhere.
It would all be more important if files were still as relevant as they used to be. I used to have ogg vorbis files and photos and everything. Nowadays, it's really only code and occasionally a document. Otherwise I'm in my web browser or on an app on my phone.
Inserting data A into B, serializing it into C, storing it as D then morph it back into C, turn it back into B again so that you can read A from it???
If A is 10 bytes, B is 10 GB OR we don't know if A exists in C? The ease of use and/or performance gain would be substantial even without memorizing 1000 manuals worth of exotic formats.