Persistent “pipes” in Linux (gist.github.com)

Soooo...

What happens when an errant grep wanders in to the directory holding these persistent pipes?

I wonder how it behaves in terms of performance and disk fragmentation. Potentially log-oriented databases could adopt that technique instead of having lots of files on disk.

Some previous thoughts: https://news.ycombinator.com/item?id=10068591

See also Varnish's log implementation, which employs a similar technique but uses a shared memory region instead: http://varnish-cache.readthedocs.org/en/latest/users-guide/o...

I would like to know what the file system implementors have to say about this technique. Is their reaction “Aargh, why would you do that, it wrecks all our optimizations!” or is it “Huh, OK, that’s kind of odd but I guess it would technically work” or even “Yes, finally someone who takes advantage of all our nice features like we always wanted!”?

Another approach (used by qmail and I'm sure other programs) is to use the file system as a database. Process 1 creates a file into a temp directory. Once writing is finished, the file is atomically moved to the working directory where Process 2 deals with it. That wouldn't work so well if you need them to be processed in a specific order, though.

Hmm, sparse files get a little annoying after the offset gets high enough. I've had weirdness with sparse files that had offsets in the 64bit range. So... eventually if you don't compact and start from a low offset ring buffer style, this solution will stop working after you pass enough data through the "pipe".

Yes. That means a lot of data, but it's a boundary case worth knowing about.