To be fair you would gzip the JCOF encoding in this example too.
Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for.
If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
> To be fair you would gzip the JCOF encoding in this example too.
Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k.
If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)
Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings.
This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
You're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
Depends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ is what i've been toying with recently.
For those unfamiliar, here is the interesting bit:
> rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
While Oscar Wilde would undoubtedly have something witty to say about that, I guess it's a matter of stress: if you read it as a word and assume stress on the (invisible) first vowel, it's almost inevitable: juh-cof.
Language is not very deterministic. An unknown word can remind you of more than one other word, which can influence the way you pronounce it, certainly in English where pronunciation is so irregular. That’s not a fact of course, just a possible factor.
There is a very, very small difference between placing a slight break in the middle of the acronym as J-COF or as JC-OF, because there's a major unwritten vowel sound after J in any case.
It may be easy to read and write, but few libraries implement it to the letter because the format doesn’t support extremely common use cases.
For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)
Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.
It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.
Requiring parsers to keep keys in file order also might have been a good thing.
(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
I think that's representing the keys, so if you'd have multiple types of objects in an array the key table would have multiple rows and the leading number index for each object would define what the keys for that object would be.
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.
But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.
In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.
Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.
It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip.
If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression
In my experience it should not make much or any difference.
I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.
I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).
Well, that's true. But if performance is more important to you, just skip it and go for binary formats that can be orders of magnitude more efficient.
In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.
If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.
JCOF is not usable in many of the use cases where JSON is. Here is why:
* JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines.
* Nested objects look to be difficult, especially if the schema is not known.
* JSON is very human friendly, JCOF is not.
I do think the format is better than CSV, TSV and other delimited flat record formats. JCOF appears to be able to handle objects that are not tabular, which CSV just can't really do. A replacement for JSON this is not, and storage efficiency isn't really relevant because of the difference in purpose.
It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
> BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
Sounds like XML, which is a whole other discussion :-)
Hey, JCOF author here. I didn't notice this had made it to HN before now.
* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.
* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).
* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.
I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.
Oh, I think you have something, but I don't think JSON and JCOF are comparable. I think it makes a very interesting replacement for formats where there is a header line like CSV, TSV. JCOF allows for some level of consistent nested structures (which would be nice for header/detail records which are really common in the real world) and would be so much nicer than many of the bespoke formats you see out there... and also, sooo much more efficient than XML, and without the issues that come with XML. Incidentally, these kinds of uses are where JSON really doesn't work well because you always have the potential of a mis-shaped object.
In short: nice work. Might be fun to roll an implementation or two :-)
JSON isn't mean to be efficient, but to be a good compromise between being human readable/writable, easily generated/parsed by a machine, and familiar. CBOR and BSON make more sense, to be frank, for any use case I could think of using this in.
I don't think BSON de-dupes the keys in the manner that JCOF does? But it has other nice properties. In a game I worked on we used a BSON library that didn't really "parse" the data when you loaded a file; it just kept a pointer to the original binary data and let you perform queries against it.
We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.
This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support.
or just use a compression (after which the difference between JSON and JCOF should be negligible)
I'm curious, which more compact format would you use? Because I have looked a bit and been unable to find one, both MessgePack and CBOR produce pretty large objects. CBOR with the string reference extension comes closer, but it's still bigger, and string references don't seem very well supported in the CBOR ecosystem.
68 comments
[ 0.24 ms ] story [ 138 ms ] threadAuthor mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for.
If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k.
If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)
This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
For those unfamiliar, here is the interesting bit:
> rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends
It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes
For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)
Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.
It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.
Requiring parsers to keep keys in file order also might have been a good thing.
(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.
In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.
In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.
If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.
But "easy" wins so we keep returning to CSV and JSON. ^_^
OTOH zstd [1] should be significantly more efficient, while being as fast or faster.
[1]: https://en.wikipedia.org/wiki/Zstd
* JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines.
* Nested objects look to be difficult, especially if the schema is not known.
* JSON is very human friendly, JCOF is not.
I do think the format is better than CSV, TSV and other delimited flat record formats. JCOF appears to be able to handle objects that are not tabular, which CSV just can't really do. A replacement for JSON this is not, and storage efficiency isn't really relevant because of the difference in purpose.
It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serializati...
> BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
Sounds like XML, which is a whole other discussion :-)
* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.
* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).
* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.
I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.
In short: nice work. Might be fun to roll an implementation or two :-)
There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.
They're improvements, they're experiments, they're an attempt to move forward. Somebody cares!
but Randall Munroe said it pretty well: https://xkcd.com/927/
We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.
This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support.
or just use a compression (after which the difference between JSON and JCOF should be negligible)
Do you mean you'd use a format with a schema?
I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome.
And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.
I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.