Cool! Could this be used to make on-disk DataFrames that store arbitrary Julia types? Serializing/deserializing such dataframes with JLD has given me lots of trouble (both too slow and also sometimes the files get corrupted in ways that can't be debugged.)
Thanks for the tip Jamii, and for the nice original post.
This is pretty close to what I've been doing, but since I'm storing Julia values rather than numeric data I still to need to convert my data to strings to store in "foo.csv" and then parse back from strings when reading in individual values using `loadtable("foo.csv", output="foo/", colparsers=[parserfortype1, parserfortype2, ...])`. As a parser for nested Julia types is fairly complicated (structs containing dictionaries as fields which contain nested arrays as values which contain ...), it'd be nice to go directly to the binary form used by JuliaDB without converting to/from the string intermediate representation needed by the csv format. Perhaps Blob isn't quite the abstraction needed, but if such a thing exists I'd love to know about it!
Arrow is basically a dataframe format. It's not really suitable for building data-structures. Eg one of the things Blobs is being used to implement is a Bε-tree (http://supertech.csail.mit.edu/papers/BenderFaJa15.pdf).
Blobs is less a data-format and more a replacement for C's ability to cast arbitrary memory locations to structs.
At some point (maybe still? I haven't touched Windows in over a decade) the windows registry had this: You could just load the entire registry into a buffer and then just cast that buffer pointer to the root type.
It still needs me to opt in to declaring methods as unsafe, rather than automatically surfacing my uses of unsafe. I could definitely "trick users" by not doing that and pretending it always works (it doesn't).
I suppose the real Julia trick used to make this nice is the generated function. There isn’t really anything like this in C without code generation.
Other languages with macros can do this sort of thing but one then needs to “derive blob” or something like that.
In Haskell one could probably do it with Generic (ie the compiler generates a data representation of the type; you write a function to go from this representation to your deserialisation function; and then the compiler does a crazy amount of inlining)
An alternative way to do this in an object oriented language is with a different metaclass that does the appropriate dereferencing. But this risks being slow if not compiled well
Also being able to remove all the dispatch and stack-allocate everything. Haskell would probably do a good job of that too but in python or js, even with codegen, it would be hard to avoid heap allocation. The simple examples here might fall to escape analysis, but in production code the intermediate Blob values typically cross a lot of function boundaries.
This reminds me of the OCaml "Ancient" library[1] that I wrote. It lets you have an extra heap of OCaml objects which can be stored in an mmap'd file and even shared between processes (although the sharing must be read-only and has a number of problems like everything has to be mapped at the same address which conflicts with ASLR - we didn't use pointer<->offset conversion as done in this article because it would touch every page and you'd end up with no sharing).
I originally wrote it to analyze large data sets (where "large" meant 16+ GB which was much larger than the commonly available RAM at the time).
It integrated nicely with OCaml. You could create ordinary OCaml objects then incrementally "mark" them so those objects would be moved (recursively) into the ancient heap. The objects could still be accessed as if they were ordinary OCaml objects even if they were on the ancient heap. On the other side you could mmap a previous heap and access the objects as if they were regular OCaml objects directly (with a few shortcomings - see README). Depending on your access pattern this worked well even if available RAM was much smaller than the size of the data set.
I used Ancient years ago for https://github.com/jamii/texsearch to stop the GC pauses caused by pointlessly traversing the huge immutable index. I don't know how else I would have met the latency requirements. Thanks for writing it :)
I am not OP but I'm imagining data-oriented design. So like, things are structured and then serialized - and at some other end deserialized in place ready to read cache-coherently?
Yup pretty much. Zero-copy deserialization implies one large read into a block of memory(or mmap) which forces a specific layout. From there it's straightforward to follow standard data oriented design principals.
KDB+ the database and the underlying K language do this. The entire heap and all data structures and kept so that they can be written to the wire, read, or mapped efficiently.
The fastest way to expedite a job is not to have to do it in the first place. A homoiconic language doesn't require de/serialization. So many problems disappear when you have this design.
can someone explain to me why is this on the homepage? Not being sarcastic. I genuinely want to know why is Zero-copy deserialization in Julia relevant to most of hacker new's commmunity
33 comments
[ 3.1 ms ] story [ 66.9 ms ] thread* https://github.com/ExpandingMan/Arrow.jl
* https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-ob...
Blobs is less a data-format and more a replacement for C's ability to cast arbitrary memory locations to structs.
[1] https://github.com/ipkn/dumpable
Graphics card drivers are WAY out. File systems too.
Julia offers as much safety as most languages. You would be hard pressed to find a language which doesn't let you pull something like this.
Julia wouldn't solve the 2 languages problem if there was a lot of things you would have to drop back to C to do....
Other languages with macros can do this sort of thing but one then needs to “derive blob” or something like that.
In Haskell one could probably do it with Generic (ie the compiler generates a data representation of the type; you write a function to go from this representation to your deserialisation function; and then the compiler does a crazy amount of inlining)
An alternative way to do this in an object oriented language is with a different metaclass that does the appropriate dereferencing. But this risks being slow if not compiled well
I originally wrote it to analyze large data sets (where "large" meant 16+ GB which was much larger than the commonly available RAM at the time).
It integrated nicely with OCaml. You could create ordinary OCaml objects then incrementally "mark" them so those objects would be moved (recursively) into the ancient heap. The objects could still be accessed as if they were ordinary OCaml objects even if they were on the ancient heap. On the other side you could mmap a previous heap and access the objects as if they were regular OCaml objects directly (with a few shortcomings - see README). Depending on your access pattern this worked well even if available RAM was much smaller than the size of the data set.
[1] http://git.annexia.org/?p=ocaml-ancient.git;a=blob;f=README....
I've done this in Java to do cache aware reading to pretty great success.