IMO, the answer is pretty clear here. Use Arrow with some form of compression. Most of the other options the Julia ecosystem offers are just in case you don't have the luxury of controlling your data format (ie because you are reading data from someone else).
Also, did you look at using ZSTD? In my experience, it provides a really good compression, and is pretty fast also.
Very interesting. Must be the first time I have seen JDF.jl talked about in a post. I guess the point about confidence being low is correct, since the format is not yet finalized and stable.
Also, another point is about how fast things can go. For me, developing JDF.jl was also so that I can do things fast! But the poster correctly pointed out that in this use-case, that is not a concern.
2 comments
[ 3.7 ms ] story [ 11.5 ms ] threadAlso, did you look at using ZSTD? In my experience, it provides a really good compression, and is pretty fast also.
Also, another point is about how fast things can go. For me, developing JDF.jl was also so that I can do things fast! But the poster correctly pointed out that in this use-case, that is not a concern.