Windows is not POSIX compliant anymore. Claiming that it is and linking to an article about WSL is just confusing, as most Linux distributions are not POSIX compliant either. I assume this is based on Windows NT's historical status with SfU and UNIX compliance. But Windows doesn't need POSIX compliance today. I'm pretty sure that only ever happened because the U.S. government was pushing for POSIX in an era where UNIX operating systems were far more relevant.
WSL v1 was just a POSIX wrapper that translated system calls into their NT equivalents. It ended up being scrapped for a Hyper-V based VM approach for WSL2, but it's a pretty good argument as far as practical POSIX compatibility.
From my understanding: WSL1 emulated Linux syscalls and ran Linux-compatible ELF binaries on picoprocesses. This isn't really sufficient to call it a POSIX layer. I am not aware (and can not find evidence) of Microsoft attempting to get it certified as compliant, and in order to do so it would need a compliant root image (e.g. not Ubuntu.)
Services for UNIX was a different story, but that's not what either version of WSL is.
> Microsoft attempting to get it certified as compliant
Who would bother getting POSIX, UNIX, or FIPS 140-2 certified in today's world? We've moved on from when any of those things were important. It costs time/money and what customers would Microsoft attract?
The US Gov't no longer sees those as important requirements.
Certification isn't required to be compliant, and I'd argue that very few people actually care or refer to it in the real world.
It meets the same bar that all of the other "mostly compliant" OSes listed on the POSIX wiki page meet: you can write code against a POSIX stdlib and it'll work without involving a VM as long as you stick to some subset of interfaces and behaviors.
WSL1's compliance was actually larger than most of the OSes on that list. The last time I asked Mentor Graphics which parts of POSIX they supported on Nucleus they said "tell us what APIs your application needs and we'll implement them" (anything beyond PSE51 was largely unimplemented). Zephyr is nice enough to actually document their limitations, but the support is far from comprehensive either:
WSL1 doesn't even handle userspace. It just ships Ubuntu. And certification or no certification, Ubuntu is very trivially not POSIX-compliant. Perhaps you can make it POSIX compliant, but there is no obvious way to do this; you would have to go point-by-point.
I don't think that is any better than OSes that conform to some specific subset.
Windows NT did have a POSIX subsystem to satisfy US government requirements, in addition to the win32 subsystem (and one more?). It had no relation to Linux whatsoever.
There was also a POSIX file system API available to win32 programs, but I'm not sure how that was related.
The POSIX namespace in NTFS is important for implementing things like SfU and WSL1. That's how they get proper POSIX semantics for filenames and links. Win32 programs can technically use it too, but it's an NTFS-specific feature rather than just a Win32 one as I understand it. (It can also be used on Linux when using NTFS drivers. AFAIK, it's usually the default option for new files in ntfs3g.)
My personal favorite irony is that by default “write” on windows is more posix compliant than any Linux install using ext4. If you need write atomicity, as posix guarantees, avoid ext4.
Well, Amazon is not very interested in making its APIs compatible with other cloud providers. But the whole thing is unrelated to POSIX itself. It makes just as much sense when you replace POSIX with e.g. HDMI cables.
Hard to say what Amazon is or isn’t interested in. The success of S3 is such that significant downstream client libraries are used by people who pay for S3, enough to provide a measure of api stability, simply out of customer inertia.
Sigh. Another article written, it seems, by someone who thinks that all/most software development is on the web. "Storage systems" in the cloud ... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. But this is an illusion likely caused by insufficient native development experierence.
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.
Thank you for phrasing this so much more politely than I would have.
I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.
Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".
"POSIX file system for the web" is kind of the whole deal for the company that wrote this. They make something to mount object storage as a filesystem.
This isn't a random programmer philosophising about POSIX, it's an ad for a product.
Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
> (...) leverage the lower cost and higher throughput of object storage like S3 (...)
What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
Not to mention that apparently we're supposed to compare that to storing on your local harddisk. Isn't local storage always far higher throughput than accessing something across the internet? Being able to store and access stuff on the internet as if it's local is a great idea, but the way they sell it is unconvincing and sounds like they don't really understand what they're talking about.
Hi author here, sorry I missed this post. The performance benchmarks and cost comparisons are for comparing S3 vs EBS (ext4 formatted), EFS, FSx Lustre and others within the same datacenter (i.e. LAN use case rather than WAN use case). That means if you have an EC2 instance running in, say AWS Ohio, and are comparing those storage options also within AWS Ohio, then cunoFS is both cheaper and higher throughput than those other options. It's a different story over WAN. In that case, your own local NVMe storage is going to be cheaper and generally faster that remote storage over a WAN. But that local NVMe storage (on say your solo laptop) isn't going to have anywhere near the Enterprise-grade redundancy, availability and scalability that AWS S3/Azure Blob/Storj/Wasabi/etc has.
Yeah, I use all of those. But what I can't really use is some low-rent sales copy trying to persuade me to abandon FOSS tooling in favor of yet another rent-seeking cloud solution.
>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...]
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.
So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
> in fact, in most cases, there is no filesystem at all
Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?
There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.
CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it).
But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.
Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.
If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.
One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.
I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?
Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.
Could the POSIX standard shell move that far? Or is there absolutely no hope?
> One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.
Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that
And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.
At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.
"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."
Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.
1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).
2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.
S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.
But ultimately, they're both storing files. The differences are fairly minor.
> By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel.
So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
> 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.
They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.
As far as I know, you still don't get e.g. create-if-not-exist. The object listing is now strongly consistent, but you don't get to do atomic operations on it.
Except many of those systems are implemented on top of type 1 hypervisors, running either containers or minimal runtimes, and if there is any POSIX related APIs, it is only the minimal surface as required by ISO C and ISO C++ standards.
POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.
As a much more low-level developer, I think they have a point, but not that the interface should be exactly S3.
After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.
So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.
(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)
We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.
> The sort of cloud storage systems described all run on top of POSIX APIs
I don't quite see how web development is relevant here; I think you've missed the point. Just because it's 'object storage' doesn't mean developers are reaching for React, HTML, CSS, JavaScript, etc. Object storage is a means to put huge volumes of data elsewhere besides your own hard disk/solid state drives, or a NAS running in the basement/attic; that's all. They are a means to access that data at extremely large scale, and can work with native or web apps.
More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises.
There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns.
Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions.
> There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns
Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.
"I want object storage APIs for local storage" <= completely reasonable
"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable
Why? Because all typical, existing applications written for POSIX systems use the POSIX API (usually indirectly) to interact with the filesystem and perform IO. Being able to use that vast ecosystem of existing applications seamlessly with object storage is such a common requirement that a good dozen different object storage-to-POSIX abstraction layers exist (many of them on top of FUSE).
Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.
Have you tried running Samba on top of cunoFS ? I'd be interested in performance data on that. For SMB3 access it might be better to add a Samba VFS vfs_cunoFS interface directly. You won't need to open source your client code, just the wrapper.
It reads to me more like the title is 'flamebait' if you like, and you've been baited by it; what the article actually seems, to me, to be saying is 'POSIX good, but many things use S3-like object storage, we need a POSIX-like standard there, by the way we make cunoFS a POSIX-compat layer for "S3-compatible"'.
Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.
Yet, the largest issues with networked filesystems is that POSIX forbids canceling file accesses, the lock semantics is really difficult, and cache is hard.
POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.
Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.
Tl;Dr, admittedly, but skimmed quickly and did a search for "socket", "timer" and "thread", and the text contains no mention of any. POSIX is a lot more than cloud storage. Some may not like it, but I've been thankful a couple decades ago when I was writing a network routine on Linux/AIX following POSIX specs and it then compiled and worked beautifully on Win NT after changing just two lines.
> Why don't more companies adopt ChromeOS as an option. (self.sysadmin)
> submitted 5 days ago by psuedononymoose
> Why spend your time dealing with AD, malware, ransomware, patches etc? I run an org with over 800 adults on ChromeOS and I find myself on an island. I however see a lot of questions and concerns on here that would completely be alleviated by using ChromeOS. What is the hold up? Most everything is a web app now. Honestly trying to understand besides management won't do it
Schools have gone this route, specifically for students, and it has pluses and minuses.
Offline mode is a thing, but does not work consistently. There is no conceptual distinction regarding why some apps work offline and some don’t.
Certain things that nearly every computer can do, like say print a file, become impossible.
Every app uses its own unique set of widgets and UI concepts. No two apps, even from the same company work consistently or have the same key bindings and capabilities (Gsuite is mostly consistent, but not with Gmail, and other Google apps, move off of Google and it gets much worse very quickly).
On the other hand, my kids can log in on any device and have their environment (as spartan as that might be) the way they left it. Until they get to older grades they don’t have to learn concepts like folders and files (for better or worse).
For businesses this seems awesome. Call center workers, for example, that get kiosk mode systems that run the CRM, KB, and nothing else seems great, basically like a more flexible POS system, but for every role.
To be fair, POSIX gets updated and eventually follows the implementations. Just like the newer C standards usually just officially adopt things that gcc and sometimes clang have already been doing for decades.
POSIX and web are not incompatible thanks to emscripten and the use of musl. I am creating a UNIX-like operating system running in a web browser https://www.exaequos.com
> Object storage is not a filesystem, and was developed with different use cases in mind. Unlike the filesystems POSIX is built to interact with, object storage is unstructured, making it readily scalable, including by pooling storage across different locations
My porn collection[0] is an unstructured object storage, making it scalable because it includes pooled storage across different locations.
[0] Jokes on you, I turned vanilla and offloaded all my requirements to the famous Orange-Black, unstructured, pooled, scalable site. But I have other things what still fit this definition.
> Object storage is usually accessed using RESTful HTTP APIs and is an example of how files can be interacted with in the absence of POSIX.
Somebody, tell the author about SMB and what network access, storage and file storage are all different things.
> At cunoFS, we’ve worked with a number of developers on diverse object-storage-backed solutions for high performance and big data purposes
[Even if didn't notice where this article is hosted] Ah, another pitch from a company deeply interested in you using theirs product?
Weird blog post. Yes POSIX mostly sucks and is outdated, but what has this to do with storage systems? POSIX is not some magic portability pixie dust which makes your code automatically work across platforms, especially when Windows is one of the target platforms.
It's almost always a better idea (and not much work) to write your own thin specialized glue layer between your application code and operating system APIs.
The article is about POSIX compliance (or lack thereof) of storage systems.
And the article does not imply that “POSIX … sucks”. On the contrary: the answer to the rhetoric question in the title is obviously “no”.
> It's almost always a better idea … to write your own thin specialized glue layer between your application code and operating system APIs.
Oh, definitely. But writing good abstractions that work equally well with POSIX-compliant filesystems and with object storage is basically impossible without massive trade-offs. That’s the entire point of the product that’s advertised here: it provides a link between object storage and POSIX-compliant file access that manages these trade-offs extremely well, and it allows users to use their existing glue code for POSIX without having to deal with object storage altogether.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise.)
The problem is that it's actually really hard to write you own specialized glue layer,and so most developers who do that have poor implementations with low performance and/or incompatibilities with non-AWS solutions. In the context of object storage, we've found lots of applications have tried to add S3 support, and while they work with AWS S3 and have basic functionality, they fail on a lot of other S3-compatible solutions. So they end up tied to AWS, and you can't use them on Microsoft Azure Storage, or often on Google Cloud Storage (despite its S3 gateway), or others.
For instance, a key workhorse of the genomics field is `samtools`, which works with AWS S3 in some ways, but not others (like Amazon Resource Names[1]). Our approach works across vendors transparently, and on S3, is much faster compared to such native implementations.
Unpopular opinion but I like Microsoft's API design of having types for most things and using structs and C++ classes more than the Unix/Posix design of most things being an integer handle (yes I know Win32 has HANDLE objects). For me this translates to DirectX's everything-gets-a-type being preferable to OpenGL's everything-returns-GLuint
I don't believe there's a single person on this planet that likes using ioctls on a int fd.
I'd say Vulkan is the best API that follows the Microsoft API design - the initialization process is always "fill out this struct and call vkCreateWhatever", and the struct fields are actually documented well, not "HANDLE handle - this is the handle".
Whenever I run into a situation where ioctl is necessary it makes me think the authors either hadn’t thought their design through enough, or couldn’t convince enough people that a sys call was justified. What an abhorant interface.
Which for anyone diving in VMS manuals, it is quite similar in concepts.
As for COM all over the place, specially after Vista, the idea is great, the execution from multiple C++ libraries, and poor VS tooling, since the OLE 1.0 days not so much.
VB 6 and .NET are the best developer experience for dealing with COM, but WinDev rather impose on us dealing with C++ and clunky tooling.
> this is not a recipe for stability of your binaries
I see the point in theory, but it works incredibly well in practice. The specific technology has been used in production by major companies for years. It even carefully works around buggy software that makes incorrect assumptions for undefined behaviour.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise. I do still use the product, because it’s vastly superior to the alternatives.)
Certainly it can be made to work, I spent years on things like it, etc.
I actually did similar in production at Google, which i suspect is one of the companies you are referring to.
I'm even overall a fan of using JITted code in production for normally "static" languages.
But - i've never seen this sort of thing break through the significant resistance/feeling of taboo that often exists around doing that kind of thing in production, long term.
You will find companies here and there willing, sure. But writ large? It eventually goes away, even at companies willing to try it.
I do hope they get past all that, i just ... am skeptical.
Yes we have many large companies (Fortune Global 500) down to small organisations using our software with this kind of interception (see for example https://cuno.io/about-us/). It took us a decent sized team a lot of years to get right, because it is so very hard a problem to crack. But we think it is worth it. And for those who don't want to use such interception, we do offer a FUSE layer as well that still offers much higher performance than alternatives.
> … at Google, which i suspect is one of the companies you are referring to.
Just to clarify: no, I was referring to companies which are using products from the company behind cunoFS, and which share the actual code base of the functional interposition. My point is that while this technique is complex and brittle in general, this specific code-base is incredibly battle-tested and has proved itself even in fairly arcane configurations.
You’re definitely right about there being some amount of resistance, but functional interposition offers some compelling advantages over all alternative solutions in terms of ease of use and unparalleled performance.
I can't entirely agree with this, but I like how the authors think about the issue (S3 being a de-facto standard without anything formal to back it up).
POSIX is outdated and problematic. It's outdated because it was conceived when hardware was vastly different than today. Storage used to be orders of magnitude slower than compute. Today, it's the opposite. POSIX APIs make squeezing all the juice out of the underlying hardware impossible. A standard duplex 100Gbps network link will carry ~300M small IP frames per second. You'll be lucky if you can do more than a few 10k/s using POSIX APIs. That's a massive bottleneck.
And it's not something that can be fixed with clever implementations. None of the POSIX APIs are async. So, to drive concurrency, programmers have to resort to threads that don't scale. That's a fundamental issue that no hardware improvement and/or software trickery will ever fix.
Today's reality is that software is seldom written against POSIX but against Linux, which offers many more APIs. Linux is, likewise, not formally standardized. It dodges the problem of S3 because it's open-source and ubiquitous. But that's not a good solution: it stifles innovation. For that reason, there hasn't been any new (production-ready, serious) kernel in decades.
We are at a deadlock, and POSIX is part of the problem.
That's funny. I was listening to this video (https://www.youtube.com/watch?v=bzkRVzciAZg&t=1s) Node.js is badass rockstar tech just barely, and one of the exact quotes is "Threads don't scale". I assumed it was just a joke, but here was see Poe's law in full effect.
I think there are a few misunderstandings there (the video is funny, btw). The kind of async IO I'm referring to exists one layer below whatever Apache and node are using. Both are event-driven, actually. They both use some flavor of epoll (see https://httpd.apache.org/docs/2.4/mod/event.html).
Historically, buffered IO was sufficient to circumvent slow threads. Simply because buffered IO operations usually don't block (you merely memcpy, and the kernel flushes asynchronously in the background). That can only take you so far, however, and it's apparent today that hardware can go much further and that the gap is widening.
It's a valid point, though, to question whether that additional performance is even needed. John Ousterhout (https://www.youtube.com/watch?v=o2HBHckrdQc) is currently working on a new network protocol to alleviate some of the problems TCP creates in software wrt performance, and he also questions whether there is even a need for very fast networking for real-world applications.
IMO, the mere existence of stuff like DPDK is proof enough. Many folks use it and would rather use the kernel if it could provide comparable performance.
> “Over the years, we’ve done lots of nice “extended functionality” stuff. Nobody ever uses them. The only thing that gets used is the standard stuff that everybody else does too.” — Linus Torvalds
Linus’s quote in the article isn’t really true. It’s true in that small, incremental changes are often ignored in the interest of portability (I’d rather my app worked on both macos and linux so will use generic facilities when at all possible; the system-specific ones may be easier and better but I’ll just have to write something different for the other system.
But when the change is a big one, the calculus is different. Consider io_uring: it’s a big win so worth writing special code to use. But the trend in these cases is to write a higher level library , or modify an existing library to work with io_uring. Not a lot of people use system calls directly, except perhaps `stat` and a few other filesystem calls like that.
Another case is the procfs, because you can use it directly from the shell.
Kinda Content marketing for another POSIX on S3 product, though I like the API interception vs FUSE approach. I'm curious how you get performance with object store latency without sacrifices unless you're intermediating. Other solutions typically run at least a fast metadata service.
Yes, we've had to do a lot of things to deal with object storage latency. Since cunoFS is running inside a given process itself, it has greater visibility into what actions it can take and is likely to take. This means we can make huge improvements in our prediction logic, so that we can prefetch within and across files much better, thus hiding latencies. For POSIX metadata, we have a caching mechanism which is shared between processes, and we've invented a better way to encode this metadata so that it is retrieved alongside the LIST operation.
The problem with POSIX file APIs are the semantics in the presence of multiple concurrent writers. Object storage (kind of) gets around this because you can't really have multiple concurrent writers to the same object.
Another classic example is that POSIX does not specify what happens when one program opens a directory stream and starts calling readdir/seekdir and then another program adds entries to the directory. Does the first program get entries in its stream for those members? Who's to say, that's up to the file system implementer.
I don't actually know if S3 guarantees that if a new stream is created with ListBuckets/ListObjects that the stream will return any new bucket/object created before the stream is closed by the client. Are those semantics defined, or is it left up to the implementer too?
> The problem with POSIX file APIs are the semantics in the presence of multiple concurrent writers.
That sure is a problem with the file APIs, but far from the only one. Other issues include weird asynchrony, weird scatter/gather, lack of standard control over exposing parallelism to the device, broken fsync() and fdatasync() error semantics, just enough default buffering features to cause confusion but not avoid most performance bugs, lack of exposing device atomic write units, and many other things.
It's an old API we've had for a long time. Made sense 40 years ago. But today it's just too low level to be easy, and too high-level to be fast.
Posix is actually pretty terrible. It is funny how Linux people still defend it. My criticism.
Inflexible.
Poor ACL management.
All this recursive bs that I still can't wrap my head around.
User groups, not a flat hierarchy?
Poor observability.
Windows is terrible in many ways but with files and in comparison to Posix Windows actually got it right.
EDIT: I did not mention the article. It is silly to compare S3 to Posix. However. Azure and other cloud providers that did offer Posix storage have dropped those offerings.
Yes, POSIX-like file access have plenty of rough edges but alternatives are mostly trading this for that, not straight up upgrade. I'd like to see deburred version (cutting all the edge case crap with fsyncing and such, among other problems), but good luck getting everyone on board of that
Every feature comes with drawbacks. For example ability to just lock the file might be great from app perspective but it is PITA from any distributed backend perspective. Ability to change it after write comes with similar problems so S3 and other cloudy ones tend to skip that.
ACLs will always be fucking mess because the more powerful ones you need the more messy access control becomes. And it always becomes married with whatever underlying system uses for authentication.
Like, you can rewrite app to use S3 but suddenly you now can't rewrite header of the file without going full read-modify-write cycle because modifying an object would be hard at scale so Amazon decided to just not do it.
So stream-encoding MP4 file directly to S3 (which at end of encoding requires writing some stuff in header) can't be done directly on S3 and you need a temporary file (guess in what? POSIX local filesystem) before the upload, or use different format.
> Poor observability.
What that has to do with anything ? It's just API, you can observe it as any other API... hell, it's actually far more observable than app using S3 as you can't strace that but you can strace filesystem operations.
Funny you should give the stream-encoding MP4 example, because yes that is what people's experience has been for S3. We've solved that - no temporary local file needed - all streamed directly to S3, for example using ordinary ffmpeg. The trick is a deeper understanding of how multi-part upload works, and if necessary, server-side copy semantics on those parts.
Depends on what you mean by metadata. MP4 metadata is data inside the file - and is modified by server-side-copy semantics that replaces only the bits that are changed. If you mean POSIX metadata, we avoid storing that in the object, and for performance store that elsewhere (it's encoded and compressed in the actual filename of hidden files).
Assertion: You can’t just put a POSIX API over cloud storage, because the semantics are fundamentally different. You can translate the POSIX API operations to S3 operations, but the client author will have to be completely aware of the translation, it can’t possibly be transparent to them except in very limited circumstances.
The problems with this approach date back all the way to AFS in the 80s, which was a full-file transfer with a local cache much like an S3 layer has to be. Simple example: it was immediately discovered that nobody ever checks the return code of close(). In a network upload situation you can definitely get an error from close() and you have to deal with it.
Yes agree that you can't "just" put a POSIX API on S3, but that doesn't make it impossible. For the sake of keeping the article to a reasonable size, I left a lot of things out. There are tradeoffs that occur between POSIX semantics, consistency and performance. Each application/process has different needs but the great thing about running right inside the process is that we can see what those needs are and adapt. For example, many applications have no need for random access writes - the only libc calls and syscalls exposed are purely sequential. Some processes have both random access writes and POSIX record locks around them to protect them from other concurrent processes - and we can see that. That means we treat these applications/files differently, with some corresponding performance implications. This is very different to a normal filesystem that has to treat every process the same way because it is a "black box".
You're right that AFS, and for that matter NFS, can in principle return error on close which many existing applications unfortunately aren't written to handle. However, that doesn't mean that NFS isn't practical - it is very widely used.
Our customers mostly run workloads in the same region as the object storage (whether in cloud or on-prem) typically with very high availability. As an essentially networked file system, you're right that it can't make much stronger guarantees than the NFS protocol itself does, but operating inside cloud infrastructure you typically see 4 9s availability.
I’m sure you’re aware of the issues, and this looks like highly useful work! But I’ve seen enough human nature to know that a lot of people will see “POSIX API” and assume they can just run anything on it without further thought. I know this because for years I’ve seen people run things on AFS and NFS, see weird concurrency behavior or data loss or latency, and blame the filesystem for not performing miracles, rather than blaming the application for not taking nonlocal storage into account.
The strongest argument for using a different API for object storage is that you don’t get that excuse. The API presents the true semantics and failure conditions and the application needs to think them through. (And your position that it may not be a strong enough argument is perfectly valid.)
POSIX was outdated at its inception. But it turns out that's fine. Lots of people have gotten plenty of good use out of POSIX. It does basic things in a straightforward way and it's easy to implement.
Cloud-based systems are basically the new UNIX, in that they are all completely different and non-standard and it's impossible to write something portable for more than one cloud provider. It would be great to have a "new POSIX" designed for the Cloud era.
120 comments
[ 3.4 ms ] story [ 87.7 ms ] threadServices for UNIX was a different story, but that's not what either version of WSL is.
Who would bother getting POSIX, UNIX, or FIPS 140-2 certified in today's world? We've moved on from when any of those things were important. It costs time/money and what customers would Microsoft attract?
The US Gov't no longer sees those as important requirements.
It meets the same bar that all of the other "mostly compliant" OSes listed on the POSIX wiki page meet: you can write code against a POSIX stdlib and it'll work without involving a VM as long as you stick to some subset of interfaces and behaviors.
WSL1's compliance was actually larger than most of the OSes on that list. The last time I asked Mentor Graphics which parts of POSIX they supported on Nucleus they said "tell us what APIs your application needs and we'll implement them" (anything beyond PSE51 was largely unimplemented). Zephyr is nice enough to actually document their limitations, but the support is far from comprehensive either:
https://docs.zephyrproject.org/latest/services/portability/p...
I don't think that is any better than OSes that conform to some specific subset.
There was also a POSIX file system API available to win32 programs, but I'm not sure how that was related.
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.
I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.
Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".
This isn't a random programmer philosophising about POSIX, it's an ad for a product.
Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone
I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)
New != Good
Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.
So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
https://github.com/kahing/goofys
https://www.cuno.io/
Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?
There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs
I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?
Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.
Could the POSIX standard shell move that far? Or is there absolutely no hope?
https://frippery.org/busybox/index.html
Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that
https://lwn.net/Articles/343924/
And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.
At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.
"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."
Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.
This is why POSIX Powershell could never exist.
https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
That's completely uncalled for and obnoxious.
1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).
2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.
S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.
But ultimately, they're both storing files. The differences are fairly minor.
So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.
After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.
So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.
(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)
We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.
> The sort of cloud storage systems described all run on top of POSIX APIs
Is that a benefit or a cost?
I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.
More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises.
There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns.
Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions.
[1]: https://www.techradar.com/news/microsoft-flight-simulator-20...
Disclaimer: I am an employee at cunoFS.
Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.
"I want object storage APIs for local storage" <= completely reasonable
"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable
Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.
Hell, why do so many static pages even need javascript
Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.
You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance.
And yet, it's still a very useful thing to do.
The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful.
Hell, the extistence of terraform should make that loud and clear.
POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.
Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.
> Why don't more companies adopt ChromeOS as an option. (self.sysadmin)
> submitted 5 days ago by psuedononymoose
> Why spend your time dealing with AD, malware, ransomware, patches etc? I run an org with over 800 adults on ChromeOS and I find myself on an island. I however see a lot of questions and concerns on here that would completely be alleviated by using ChromeOS. What is the hold up? Most everything is a web app now. Honestly trying to understand besides management won't do it
https://old.reddit.com/r/sysadmin/comments/177def3/why_dont_...
Disclaimer: emphasis is mine
Offline mode is a thing, but does not work consistently. There is no conceptual distinction regarding why some apps work offline and some don’t.
Certain things that nearly every computer can do, like say print a file, become impossible.
Every app uses its own unique set of widgets and UI concepts. No two apps, even from the same company work consistently or have the same key bindings and capabilities (Gsuite is mostly consistent, but not with Gmail, and other Google apps, move off of Google and it gets much worse very quickly).
On the other hand, my kids can log in on any device and have their environment (as spartan as that might be) the way they left it. Until they get to older grades they don’t have to learn concepts like folders and files (for better or worse).
For businesses this seems awesome. Call center workers, for example, that get kiosk mode systems that run the CRM, KB, and nothing else seems great, basically like a more flexible POS system, but for every role.
Chromebooks for the call centre will come… may god have mercy on our souls.
The BSDs, Mac OS included, have ways to do things that are not in POSIX, and they're usually the best way to get that specific thing done.
Linux is the same, but doesn't follow the BSDs lead on those APIS (epoll, inotify, kqueue/kevent etc).
io_uring is awesome, but there's no analog for FreeBSD etc.
Fuchsia, Windows and other platforms offer Linux compatibility, not as focused on specifically POSIX.
So, is Linux the new POSIX? I don't know...
... but I think about this monthly at least.
https://en.cppreference.com/w/c/23
That list isn't exhaustive.
> Object storage is not a filesystem, and was developed with different use cases in mind. Unlike the filesystems POSIX is built to interact with, object storage is unstructured, making it readily scalable, including by pooling storage across different locations
My porn collection[0] is an unstructured object storage, making it scalable because it includes pooled storage across different locations.
[0] Jokes on you, I turned vanilla and offloaded all my requirements to the famous Orange-Black, unstructured, pooled, scalable site. But I have other things what still fit this definition.
> Object storage is usually accessed using RESTful HTTP APIs and is an example of how files can be interacted with in the absence of POSIX.
Somebody, tell the author about SMB and what network access, storage and file storage are all different things.
> At cunoFS, we’ve worked with a number of developers on diverse object-storage-backed solutions for high performance and big data purposes
[Even if didn't notice where this article is hosted] Ah, another pitch from a company deeply interested in you using theirs product?
It's almost always a better idea (and not much work) to write your own thin specialized glue layer between your application code and operating system APIs.
The article is about POSIX compliance (or lack thereof) of storage systems.
And the article does not imply that “POSIX … sucks”. On the contrary: the answer to the rhetoric question in the title is obviously “no”.
> It's almost always a better idea … to write your own thin specialized glue layer between your application code and operating system APIs.
Oh, definitely. But writing good abstractions that work equally well with POSIX-compliant filesystems and with object storage is basically impossible without massive trade-offs. That’s the entire point of the product that’s advertised here: it provides a link between object storage and POSIX-compliant file access that manages these trade-offs extremely well, and it allows users to use their existing glue code for POSIX without having to deal with object storage altogether.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise.)
The problem is that it's actually really hard to write you own specialized glue layer,and so most developers who do that have poor implementations with low performance and/or incompatibilities with non-AWS solutions. In the context of object storage, we've found lots of applications have tried to add S3 support, and while they work with AWS S3 and have basic functionality, they fail on a lot of other S3-compatible solutions. So they end up tied to AWS, and you can't use them on Microsoft Azure Storage, or often on Google Cloud Storage (despite its S3 gateway), or others.
For instance, a key workhorse of the genomics field is `samtools`, which works with AWS S3 in some ways, but not others (like Amazon Resource Names[1]). Our approach works across vendors transparently, and on S3, is much faster compared to such native implementations.
[1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-a...
I'd say Vulkan is the best API that follows the Microsoft API design - the initialization process is always "fill out this struct and call vkCreateWhatever", and the struct fields are actually documented well, not "HANDLE handle - this is the handle".
As for COM all over the place, specially after Vista, the idea is great, the execution from multiple C++ libraries, and poor VS tooling, since the OLE 1.0 days not so much.
VB 6 and .NET are the best developer experience for dealing with COM, but WinDev rather impose on us dealing with C++ and clunky tooling.
It does LD_PRELOAD based dynamic syscall interception - which is dangerous enough, but also doesn't work for a lot of cases.
So they have a JIT ELF translator that will redirect calls for static binaries.
While all this is cool technically, this is not a recipe for stability of your binaries.
I see the point in theory, but it works incredibly well in practice. The specific technology has been used in production by major companies for years. It even carefully works around buggy software that makes incorrect assumptions for undefined behaviour.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise. I do still use the product, because it’s vastly superior to the alternatives.)
I actually did similar in production at Google, which i suspect is one of the companies you are referring to.
I'm even overall a fan of using JITted code in production for normally "static" languages.
But - i've never seen this sort of thing break through the significant resistance/feeling of taboo that often exists around doing that kind of thing in production, long term.
You will find companies here and there willing, sure. But writ large? It eventually goes away, even at companies willing to try it.
I do hope they get past all that, i just ... am skeptical.
Yes we have many large companies (Fortune Global 500) down to small organisations using our software with this kind of interception (see for example https://cuno.io/about-us/). It took us a decent sized team a lot of years to get right, because it is so very hard a problem to crack. But we think it is worth it. And for those who don't want to use such interception, we do offer a FUSE layer as well that still offers much higher performance than alternatives.
Just to clarify: no, I was referring to companies which are using products from the company behind cunoFS, and which share the actual code base of the functional interposition. My point is that while this technique is complex and brittle in general, this specific code-base is incredibly battle-tested and has proved itself even in fairly arcane configurations.
You’re definitely right about there being some amount of resistance, but functional interposition offers some compelling advantages over all alternative solutions in terms of ease of use and unparalleled performance.
--
[1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
POSIX is outdated and problematic. It's outdated because it was conceived when hardware was vastly different than today. Storage used to be orders of magnitude slower than compute. Today, it's the opposite. POSIX APIs make squeezing all the juice out of the underlying hardware impossible. A standard duplex 100Gbps network link will carry ~300M small IP frames per second. You'll be lucky if you can do more than a few 10k/s using POSIX APIs. That's a massive bottleneck.
And it's not something that can be fixed with clever implementations. None of the POSIX APIs are async. So, to drive concurrency, programmers have to resort to threads that don't scale. That's a fundamental issue that no hardware improvement and/or software trickery will ever fix.
Today's reality is that software is seldom written against POSIX but against Linux, which offers many more APIs. Linux is, likewise, not formally standardized. It dodges the problem of S3 because it's open-source and ubiquitous. But that's not a good solution: it stifles innovation. For that reason, there hasn't been any new (production-ready, serious) kernel in decades.
We are at a deadlock, and POSIX is part of the problem.
Historically, buffered IO was sufficient to circumvent slow threads. Simply because buffered IO operations usually don't block (you merely memcpy, and the kernel flushes asynchronously in the background). That can only take you so far, however, and it's apparent today that hardware can go much further and that the gap is widening.
It's a valid point, though, to question whether that additional performance is even needed. John Ousterhout (https://www.youtube.com/watch?v=o2HBHckrdQc) is currently working on a new network protocol to alleviate some of the problems TCP creates in software wrt performance, and he also questions whether there is even a need for very fast networking for real-world applications.
IMO, the mere existence of stuff like DPDK is proof enough. Many folks use it and would rather use the kernel if it could provide comparable performance.
Linus’s quote in the article isn’t really true. It’s true in that small, incremental changes are often ignored in the interest of portability (I’d rather my app worked on both macos and linux so will use generic facilities when at all possible; the system-specific ones may be easier and better but I’ll just have to write something different for the other system.
But when the change is a big one, the calculus is different. Consider io_uring: it’s a big win so worth writing special code to use. But the trend in these cases is to write a higher level library , or modify an existing library to work with io_uring. Not a lot of people use system calls directly, except perhaps `stat` and a few other filesystem calls like that.
Another case is the procfs, because you can use it directly from the shell.
Yes, we've had to do a lot of things to deal with object storage latency. Since cunoFS is running inside a given process itself, it has greater visibility into what actions it can take and is likely to take. This means we can make huge improvements in our prediction logic, so that we can prefetch within and across files much better, thus hiding latencies. For POSIX metadata, we have a caching mechanism which is shared between processes, and we've invented a better way to encode this metadata so that it is retrieved alongside the LIST operation.
Another classic example is that POSIX does not specify what happens when one program opens a directory stream and starts calling readdir/seekdir and then another program adds entries to the directory. Does the first program get entries in its stream for those members? Who's to say, that's up to the file system implementer.
I don't actually know if S3 guarantees that if a new stream is created with ListBuckets/ListObjects that the stream will return any new bucket/object created before the stream is closed by the client. Are those semantics defined, or is it left up to the implementer too?
That sure is a problem with the file APIs, but far from the only one. Other issues include weird asynchrony, weird scatter/gather, lack of standard control over exposing parallelism to the device, broken fsync() and fdatasync() error semantics, just enough default buffering features to cause confusion but not avoid most performance bugs, lack of exposing device atomic write units, and many other things.
It's an old API we've had for a long time. Made sense 40 years ago. But today it's just too low level to be easy, and too high-level to be fast.
Inflexible. Poor ACL management. All this recursive bs that I still can't wrap my head around. User groups, not a flat hierarchy? Poor observability.
Windows is terrible in many ways but with files and in comparison to Posix Windows actually got it right.
EDIT: I did not mention the article. It is silly to compare S3 to Posix. However. Azure and other cloud providers that did offer Posix storage have dropped those offerings.
Yes, POSIX-like file access have plenty of rough edges but alternatives are mostly trading this for that, not straight up upgrade. I'd like to see deburred version (cutting all the edge case crap with fsyncing and such, among other problems), but good luck getting everyone on board of that
Every feature comes with drawbacks. For example ability to just lock the file might be great from app perspective but it is PITA from any distributed backend perspective. Ability to change it after write comes with similar problems so S3 and other cloudy ones tend to skip that.
ACLs will always be fucking mess because the more powerful ones you need the more messy access control becomes. And it always becomes married with whatever underlying system uses for authentication.
Like, you can rewrite app to use S3 but suddenly you now can't rewrite header of the file without going full read-modify-write cycle because modifying an object would be hard at scale so Amazon decided to just not do it.
So stream-encoding MP4 file directly to S3 (which at end of encoding requires writing some stuff in header) can't be done directly on S3 and you need a temporary file (guess in what? POSIX local filesystem) before the upload, or use different format.
> Poor observability.
What that has to do with anything ? It's just API, you can observe it as any other API... hell, it's actually far more observable than app using S3 as you can't strace that but you can strace filesystem operations.
Funny you should give the stream-encoding MP4 example, because yes that is what people's experience has been for S3. We've solved that - no temporary local file needed - all streamed directly to S3, for example using ordinary ffmpeg. The trick is a deeper understanding of how multi-part upload works, and if necessary, server-side copy semantics on those parts.
The problems with this approach date back all the way to AFS in the 80s, which was a full-file transfer with a local cache much like an S3 layer has to be. Simple example: it was immediately discovered that nobody ever checks the return code of close(). In a network upload situation you can definitely get an error from close() and you have to deal with it.
Yes agree that you can't "just" put a POSIX API on S3, but that doesn't make it impossible. For the sake of keeping the article to a reasonable size, I left a lot of things out. There are tradeoffs that occur between POSIX semantics, consistency and performance. Each application/process has different needs but the great thing about running right inside the process is that we can see what those needs are and adapt. For example, many applications have no need for random access writes - the only libc calls and syscalls exposed are purely sequential. Some processes have both random access writes and POSIX record locks around them to protect them from other concurrent processes - and we can see that. That means we treat these applications/files differently, with some corresponding performance implications. This is very different to a normal filesystem that has to treat every process the same way because it is a "black box".
You're right that AFS, and for that matter NFS, can in principle return error on close which many existing applications unfortunately aren't written to handle. However, that doesn't mean that NFS isn't practical - it is very widely used.
Our customers mostly run workloads in the same region as the object storage (whether in cloud or on-prem) typically with very high availability. As an essentially networked file system, you're right that it can't make much stronger guarantees than the NFS protocol itself does, but operating inside cloud infrastructure you typically see 4 9s availability.
The strongest argument for using a different API for object storage is that you don’t get that excuse. The API presents the true semantics and failure conditions and the application needs to think them through. (And your position that it may not be a strong enough argument is perfectly valid.)
Cloud-based systems are basically the new UNIX, in that they are all completely different and non-standard and it's impossible to write something portable for more than one cloud provider. It would be great to have a "new POSIX" designed for the Cloud era.