The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad.
Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero).
Access is read only.
To be clear, this is a test rig, not for actual data storage.
Some comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally.
Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographically secure hash is worth the extra computational cost to users.
Hi, the author here. Thanks for taking a look at the code!
I picked SHA-1 because it was the simplest way to generate pseudorandom data: I knew there's a Rust crate for it which I used before. Works fine for a proof-of-concept :) I'm now reading up on other hashes and ciphers. My tentative plan was to replace SHA-1 with AES-NI in counter mode, but if I can find something that works on all CPUs while giving me a comparable performance, I'll go for that. (Should probably look at PRNGs as well; if I concatenate the file seed with the counter and use that as a seed for a PRNG, then ask it for a value -- it should be good enough too.)
I doubt Lars has requirements for the quality of randomness, but we'll see.
ndb is also quite fun in that you can rate-limit easily, so you can check what happens with your app on slow storage. don't make it too slow, the Linux kernel doesn't like that and can hang completely.
If I'm understanding correctly, this still requires one instance with real (vs. virtual) storage on it? The advantage of PlentyFS is that it effectively requires no storage anywhere.
I didn't write PlentyFS, but was participating (mostly not helpfully) in Lars Wirzenius's thread on Mastodon.
My idea was to look at an HTTP or WebDAV solution using FUSE. The server would "create' the content (thinking along the line of HTTP honeypots), the FUSE module would present the filesystem-based access. There's an existing WebFS fuse modele, also a WebDAV one IIRC.
Unfortunately, HHTP doesn't offer directory presentation, making full filesystem semantics difficult.
I'm looking at other possible options as well, the concept is interesting.
9 comments
[ 3.3 ms ] story [ 13.7 ms ] threadhttps://en.wikipedia.org/wiki/The_Library_of_Babel
Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero).
Access is read only.
To be clear, this is a test rig, not for actual data storage.
Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographically secure hash is worth the extra computational cost to users.
I picked SHA-1 because it was the simplest way to generate pseudorandom data: I knew there's a Rust crate for it which I used before. Works fine for a proof-of-concept :) I'm now reading up on other hashes and ciphers. My tentative plan was to replace SHA-1 with AES-NI in counter mode, but if I can find something that works on all CPUs while giving me a comparable performance, I'll go for that. (Should probably look at PRNGs as well; if I concatenate the file seed with the counter and use that as a seed for a PRNG, then ask it for a value -- it should be good enough too.)
I doubt Lars has requirements for the quality of randomness, but we'll see.
As in this article, we also use this for testing.
If I'm understanding correctly, this still requires one instance with real (vs. virtual) storage on it? The advantage of PlentyFS is that it effectively requires no storage anywhere.
I didn't write PlentyFS, but was participating (mostly not helpfully) in Lars Wirzenius's thread on Mastodon.
My idea was to look at an HTTP or WebDAV solution using FUSE. The server would "create' the content (thinking along the line of HTTP honeypots), the FUSE module would present the filesystem-based access. There's an existing WebFS fuse modele, also a WebDAV one IIRC.
Unfortunately, HHTP doesn't offer directory presentation, making full filesystem semantics difficult.
I'm looking at other possible options as well, the concept is interesting.