Precisely. Many people don't realise how exceptionally well JSON compresses. Provided you're using it the way most do, to send arrays of objects which share the same set of keys (or some subset thereof), then all the keys will end up dictionary-coded away, thus totally eroding the space advantage that this format notionally has.
Plus JSON's exceptionally wide support means you can benefit from SIMD-assisted decoders which will absolutely blow this out of the water – and much, much more besides. I wish people would devote their time to something more useful than 'yet another competing standard'.
Edit: Sorry, I want to be clear, this is an impressive and cool personal project. I hope it's a step on an exciting journey for the person who wrote it. It just doesn't actually have enough strengths to replace JSON - which would be a tall order for any new format.
Looks neat. I don't see a formal spec. Question: if I have two optional fields of the same type and the first one isn't provided, how does a parser know which field is provided? The optional fields seem unclear to me.
I'm sceptical about the value proposition of this without seeing much more than a simple example that offers little over existing hypermedia+json/csv practices.
If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) but you don't want to give up the ability to include metadata in your JSON, there are a ton of different ways for structure your document to solve this issue without inventing new document formats.
Also this example is unclear (possibly ambiguous?); how is "int" as a type for the "age" column distinguished from "street", "city", etc as what I assume are field names?
> If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for)
Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it. The only case in which that wouldn't be true is when dealing with extremely unusual and heteromorphic data, but then this format doesn't seem to support such data at all.
I'm also mystified that the author claims this is readable. It looks eminently unreadable compared with JSON, if you have anything beyond one row of very simple data with all optional fields present. And, in that case, it's basically just 'JSON with the keys on a different row'.
(Congrats to the author, but this is more of a fun personal project rather than something to seriously present as a 'JSON killer'. If you do present it as a JSON killer, then you have to expect a rigorous review.)
> Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it.
Gzipping indeed helps in getting mostly back the space taken by the field names, but a parser will still have to parse these strings. On a large document, this might have a performance impact.
One good side of having the field names however is that one can reorder them adlib.
That's true, but the main argument made by the website is about the space advantage, so it's very relevant that that space advantage is basically nullified by the widespread use of compression.
If your worry is parsing speed, then JSON not only has battle-tested parsers, but also has SIMD-assisted parsers which can process gigabytes a second on a single core (e.g. https://github.com/simdjson/simdjson). It would take Internet Object years to develop parsers as performant as that, even if it did, by some miracle, achieve wide uptake. So the notional advantage afforded by not having keys on each row is neither here nor there.
And incidentally, as someone who's written a handful of parsers, I suspect that this scheme would not be particularly easy to parse. You need lookahead because of optional fields, as well as maintaining state and a lookup table for mapping positions to keys, etc. I can draw up a quick parser in pseudocode or Python to explain, if you disagree.
I am not personally worried by perf in either case, but I see your point.
> It would take Internet Object years to develop parsers as performant as that
Well, implementing a JSON parser is arguably difficult, for many reasons, I suppose the main one is the flexibility it provides. I don't know if this would be the case for this format however. TBH, I doesn't seem to add too much to CSV, and perhaps it would be simpler to use CSV with the first line of this format has a hint for the data structure.
I agree. CSV + Metadata/field types (which JSON can handle) plus zipping (dictionary coding) takes care of, what, 99.9999% of the issues folks have with one type or the other?
As far I can see "IO" addresses the size issue, which is indeed a compression issue for the most part.
For a broader take on an alternative, there is concise encoding Concise Encoding [1][2], which I believe addresses a few more issues with existing encodings (clear spec, schema not an afterthought, native support for a variety of data structures, security, ...).
People keep saying “just use gzip and JSON is plenty small” but gzip isn’t free. It takes time and power to do all the compression and decompression. The uncompressed size of the data takes up memory on client and server.
A smaller data format requires less compression time and power and you can fit more of it in memory at either end.
Since strings don't need to be quoted, what happens during deserialization if you want the string "T"? Does this lead to the equivalent of the Norway-Problem of YAML [0]?
Is the space between the key and the type necessary? If not, how to distinguish between objects and types?
Does the validation offer some form of unions or mutual exclusion?
YAML and its "Arrays" are really broken. The problem I see with Internet Object is that it's also implying this kind of mechanism.
Every time I read about new formats, they seem to get either the 1-n relations or the n-n relations implemented well, but not both. I guess that's what's so hard about map/reduce...
Regarding YAML: somebody on HN mentioned his project DIXY a couple years ago, and it's much much _much_ easier to parse than YAML. [1] I'm using this over YAML pretty much everywhere now.
That there are multiple ways to define Arrays: "- item", "-\n\titem", "\titem" or "item, item" for starters. Parsing YAML into Arrays requires context of its surroundings.
Without the previous context, you cannot know what type of data you're parsing when you are at a "-" at the beginning of a line or a "," in the middle of a line.
This is just unnecessary parser complexity and human ambiguity in my opinion.
As a question to you in case you disagree: What happens when you write down an indented/nested "\t- name: John, Doe"? It's pretty much unpredictable without the previously parsed data structures or their history in YAML.
(I don't wanna start the discussion of "<<" and how it influences the parsing context of YAML data structures. I think the merge key also has no place in a data serialization format.)
Agreed. YAML does have some use cases. I find it useful when I want to manually write lots of JSON data for test scripts. But the format, because it tries to be concise, ends up to be hard to manually parse.
I don't consider YAML a good serialisation format.
Yaml has so many problems. Python 3.10 raised a new one to my attention when the core devs realized their arrays of versions contained twice 3.1 and no 3.10. Indeed, if write unquotted ascii, yaml gives you strings. Except if it can cast it to a number that is.
TOML is better, but it still has more gotchas that necessary. So much I find it easier to just edit a python file
I'm thinking of giving a try to cue. Any feedback ?
Dixy looks easy, but "There is only one simple rule. In Dixy, everything is a dictionary [string:string]" isn't accurate or helpful.
It's also [string:dictionary] and [string:?] where ? means nil. White space matters, and tab is fixed at 4 spaces wide. When creating text from a dictionary it adds "# Dixy 1.0\n\n" which means loading and saving will change the file every time! Not sure what other issues there are, but I noticed this line:
// TODO: if key is numeric, parse as Array
It does look simple though. It'd be nice if someone made strict rules and addressed the corner cases.
It seems to be a typed CSV, so whether `T` is interpreted as a string or a boolean presumably depends on the schema. That sounds slightly better than YAML, though it can easily break when you allow heterogeneous types (say, string or boolean).
Json is a good format to represent results of aggregation queries (group by in sql) using nesting and storing data in a single file.
Without that you would need to either
1. store multiple not-nested (tabular, eg. csv) files and join them at the time of use.
2. denormalize all these csvs into a single big csv duplicating the same values over and over. Compression should handle this at storage time, bht you still pay the cost when reading.
3. store values by columns, not by rows, adding various RLE and dict encodings to compress repeated values in columns, making the files not human friendly
4. once you store it in columns and make it unreadable, just store it as binary instead of text. You get parquet
Json and csb are simple and for that reason they won and will stay with us no matter how hard you try to add features to it.
That said I think adding a trailing comma and comments to json wouldn't be a big stretch.
The battle will be for the best columnar binary format. Parquet is the closest to a standard, but it seems to be used only as a standard for a storage. Big data systems still uncompress it and work with their own representation. The holy grail is when you get a columnar format which is good enough that big data systems use it as their underlying data representation instead of coming up with their own. I suspect such format will come from something like open sourced Snowflake, Clickhouse, Chaossearch or something like that, which has battle tested performant algorithms on them, instead of designed by committee, such as parquet.
The problem with Apache Arrow and Parquet is that you have two - one for storage and one for computation - but in the end you only want one for both. You want to run fast algorithms on memory mapped compressed columns. Not doing this stupid deserialization from parquet to arrow.
Parquet and arrow are designed by committee and try to accomplish too much for that matter. While that's good for some cases, my prediction is that there will exist a data processing system in the future whose file format will support that and be good enoigh for most data intensive applications. It will not be feature complete, like json, but will be good enough. Some devs from then on will complain about adding this and that feature to that format, but majority will be happy as they are now with json. Such format can only come from industry, not from a committee.
Right. That's why I am more interested in arrow than parquet. Going from a pure compressed storage format to incorporate computation would be more difficult than going from memory-mapped / computation format to long-term storage. Arrow already made some good choices regarding data exchange over wire, these are translatable to data exchange over time.
Of course, I am only dealing with a few hundreds GiB data, not sure at larger scale whether arrow fails.
> That said I think adding a trailing comma and comments to json wouldn't be a big stretch.
Sadly, json's designers suffered from the same hubris as the designers of markdown and gemini, when they decided to not include a version number in the file format. So you are kind of hosed if you want to make a change like that.
Before json there was xml (ugh), but before xml there were Lisp S-expressions, which seem to have handled all these issues perfectly well 50 years ago. Yet we keep re-inventing them. Greenspun's tenth law is still with us.
It's just a matter of parser implementation. These changes are backwards compatible. If python decided to add support for comments and trailing commas in json.loads, that would become the new standard, at least for data scientists, not for web devs. All the other ones would then follow.
Now whatever generates your data has to know what parser is going to read the data. The parser can't tell right away whether the data has those trailing commas. They are optional, so they might not start appearing until after gigabytes of output have gone by. So you can't count on a quick error message in the event of a version mismatch.
If you have gigabytes of handwritten JSON (if it's not handwritten, trailing vs non-trailing commas surely don't matter), then I feel like you're doing something wrong.
Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]? Are you living under a rock!?" etc etc ;)
It's silly to not write your software to handle every possible input instead of every input you think is likely based on some predictions about humans. Failure to do that is why YAML is so broken.
JSON isn't a format conducive to handwriting even if it probably should have made more accomodations for that at the start. Right now it can't even handle trailing newlines. But if you want to fix that, call it something different (maybe even JSON2), for heaven's sake.
I doubt anyone would handwrite an entire gigabyte JSON document, but they might hand-edit a machine-generated one to make a change someplace in it, end up putting in a trailing comma, and have the document pass their local tests but crash a remote parser.
> Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]?"
> they might hand-edit a machine-generated one to make a change someplace in it
Ah, there's [stupendously niche use case] ;)
Seriously, though, I do agree with your point that good software should handle every edge case. I'm not arguing that.
But the case for having trailing commas does seem to be generally predicated on handwritten JSON, so I'm saying it's _unlikely_ it would be used in that way, and therefore that such failures would be rare and thus not a very grave counterargument.
You also have to think about security implications of a hostile client being able to crash a remote parser with gigabytes of context, though if it crashes on malformed json then they could do that with a much shorter fragment. In the well intentioned case, being paged in the middle of the night to debug a weird crash isn't fun, so it's better to have an accurate diagnosis (version mismatch) in the log if not prevent the crash outright. There is also the matter of every weird javascript implementation that would have to be updated, if JS can't already accept the trailing commas. It just doesn't seem worth hassling over.
Compression and decompression (gzip) takes computing power and RAM. The resulting JSON (in memory) is still harder to parse because of the required field names...
> age:{int, min:20},
address: {street, city, state}
Unless the space after the colon is significant it seems we have to just "know" that int introduces a type definition instead of a structure.
Also
> Schema Details
JSON doesn't have built-in schema support!
seems a little disingenuous. JSON provides a name for each type of value, so there is mostly no need for the schema when viewing the data. There is a JSON Schema definition.
It’s part of the schema part, not the serialization part right? I don’t disagree you parse then validate, but in a schema that defines type of data, it’s not unreasonable to put limits on values.
If you follow that link which says "Read the Story Here" they have this json example which has a list of employees and then info about the pagination of that list. The caption is this
>If you look closely, this JSON document mixes the data employees with other non-data keys (headers) such as count, currentPage, and pageSize in the same response.
But they don't explain at all how Changing the data format fixes the underlying issue of mixed concerns in one data object.
82 comments
[ 3.9 ms ] story [ 150 ms ] threadPlus JSON's exceptionally wide support means you can benefit from SIMD-assisted decoders which will absolutely blow this out of the water – and much, much more besides. I wish people would devote their time to something more useful than 'yet another competing standard'.
Edit: Sorry, I want to be clear, this is an impressive and cool personal project. I hope it's a step on an exciting journey for the person who wrote it. It just doesn't actually have enough strengths to replace JSON - which would be a tall order for any new format.
If a compact columnar representation is what you're after to avoid having to repeat every field name in an array of objects (which CSV is good for) but you don't want to give up the ability to include metadata in your JSON, there are a ton of different ways for structure your document to solve this issue without inventing new document formats.
Also this example is unclear (possibly ambiguous?); how is "int" as a type for the "age" column distinguished from "street", "city", etc as what I assume are field names?
Plus, as I wrote elsewhere, gzipping your JSON will result in essentially "avoiding having to repeat every field name" by dictionary coding it. The only case in which that wouldn't be true is when dealing with extremely unusual and heteromorphic data, but then this format doesn't seem to support such data at all.
I'm also mystified that the author claims this is readable. It looks eminently unreadable compared with JSON, if you have anything beyond one row of very simple data with all optional fields present. And, in that case, it's basically just 'JSON with the keys on a different row'.
(Congrats to the author, but this is more of a fun personal project rather than something to seriously present as a 'JSON killer'. If you do present it as a JSON killer, then you have to expect a rigorous review.)
Gzipping indeed helps in getting mostly back the space taken by the field names, but a parser will still have to parse these strings. On a large document, this might have a performance impact.
One good side of having the field names however is that one can reorder them adlib.
If your worry is parsing speed, then JSON not only has battle-tested parsers, but also has SIMD-assisted parsers which can process gigabytes a second on a single core (e.g. https://github.com/simdjson/simdjson). It would take Internet Object years to develop parsers as performant as that, even if it did, by some miracle, achieve wide uptake. So the notional advantage afforded by not having keys on each row is neither here nor there.
And incidentally, as someone who's written a handful of parsers, I suspect that this scheme would not be particularly easy to parse. You need lookahead because of optional fields, as well as maintaining state and a lookup table for mapping positions to keys, etc. I can draw up a quick parser in pseudocode or Python to explain, if you disagree.
I am not personally worried by perf in either case, but I see your point.
> It would take Internet Object years to develop parsers as performant as that
Well, implementing a JSON parser is arguably difficult, for many reasons, I suppose the main one is the flexibility it provides. I don't know if this would be the case for this format however. TBH, I doesn't seem to add too much to CSV, and perhaps it would be simpler to use CSV with the first line of this format has a hint for the data structure.
Of these two, I like neither, but would opt for the latter.
I also considered that min:20 implied the previous had to be a type, but I don’t see how that’s consistent with
andI'd be more interested to know about serialisation and deserialisation time.
For a broader take on an alternative, there is concise encoding Concise Encoding [1][2], which I believe addresses a few more issues with existing encodings (clear spec, schema not an afterthought, native support for a variety of data structures, security, ...).
[1] https://concise-encoding.org/ [2] The author gave a presentation on it here: https://www.youtube.com/watch?v=_dIHq4GJE14
A smaller data format requires less compression time and power and you can fit more of it in memory at either end.
Is the space between the key and the type necessary? If not, how to distinguish between objects and types?
Does the validation offer some form of unions or mutual exclusion?
[0]: https://hitchdev.com/strictyaml/why/implicit-typing-removed/
Every time I read about new formats, they seem to get either the 1-n relations or the n-n relations implemented well, but not both. I guess that's what's so hard about map/reduce...
Regarding YAML: somebody on HN mentioned his project DIXY a couple years ago, and it's much much _much_ easier to parse than YAML. [1] I'm using this over YAML pretty much everywhere now.
[1] https://github.com/kuyawa/Dixy
That there are multiple ways to define Arrays: "- item", "-\n\titem", "\titem" or "item, item" for starters. Parsing YAML into Arrays requires context of its surroundings.
Without the previous context, you cannot know what type of data you're parsing when you are at a "-" at the beginning of a line or a "," in the middle of a line.
This is just unnecessary parser complexity and human ambiguity in my opinion.
As a question to you in case you disagree: What happens when you write down an indented/nested "\t- name: John, Doe"? It's pretty much unpredictable without the previously parsed data structures or their history in YAML.
(I don't wanna start the discussion of "<<" and how it influences the parsing context of YAML data structures. I think the merge key also has no place in a data serialization format.)
Agreed. YAML does have some use cases. I find it useful when I want to manually write lots of JSON data for test scripts. But the format, because it tries to be concise, ends up to be hard to manually parse.
I don't consider YAML a good serialisation format.
TOML is better, but it still has more gotchas that necessary. So much I find it easier to just edit a python file
I'm thinking of giving a try to cue. Any feedback ?
It's also [string:dictionary] and [string:?] where ? means nil. White space matters, and tab is fixed at 4 spaces wide. When creating text from a dictionary it adds "# Dixy 1.0\n\n" which means loading and saving will change the file every time! Not sure what other issues there are, but I noticed this line:
It does look simple though. It'd be nice if someone made strict rules and addressed the corner cases.- JSON - TOML - CSON - INI - ENO - XML
I like CSV for tabular data obviously. This looks, as others have mentioned, like CSV with better metadata.
I like INI for its simplicity. JSON is good for more complicated data, but I have to say I like CSON.
Without that you would need to either
Json and csb are simple and for that reason they won and will stay with us no matter how hard you try to add features to it.That said I think adding a trailing comma and comments to json wouldn't be a big stretch.
The battle will be for the best columnar binary format. Parquet is the closest to a standard, but it seems to be used only as a standard for a storage. Big data systems still uncompress it and work with their own representation. The holy grail is when you get a columnar format which is good enough that big data systems use it as their underlying data representation instead of coming up with their own. I suspect such format will come from something like open sourced Snowflake, Clickhouse, Chaossearch or something like that, which has battle tested performant algorithms on them, instead of designed by committee, such as parquet.
The problem with Apache Arrow and Parquet is that you have two - one for storage and one for computation - but in the end you only want one for both. You want to run fast algorithms on memory mapped compressed columns. Not doing this stupid deserialization from parquet to arrow.
Parquet and arrow are designed by committee and try to accomplish too much for that matter. While that's good for some cases, my prediction is that there will exist a data processing system in the future whose file format will support that and be good enoigh for most data intensive applications. It will not be feature complete, like json, but will be good enough. Some devs from then on will complain about adding this and that feature to that format, but majority will be happy as they are now with json. Such format can only come from industry, not from a committee.
Of course, I am only dealing with a few hundreds GiB data, not sure at larger scale whether arrow fails.
Sadly, json's designers suffered from the same hubris as the designers of markdown and gemini, when they decided to not include a version number in the file format. So you are kind of hosed if you want to make a change like that.
Before json there was xml (ugh), but before xml there were Lisp S-expressions, which seem to have handled all these issues perfectly well 50 years ago. Yet we keep re-inventing them. Greenspun's tenth law is still with us.
Though I'm sure someone's going to step in and say "Have you not heard of [stupendously niche use case]? Are you living under a rock!?" etc etc ;)
JSON isn't a format conducive to handwriting even if it probably should have made more accomodations for that at the start. Right now it can't even handle trailing newlines. But if you want to fix that, call it something different (maybe even JSON2), for heaven's sake.
I doubt anyone would handwrite an entire gigabyte JSON document, but they might hand-edit a machine-generated one to make a change someplace in it, end up putting in a trailing comma, and have the document pass their local tests but crash a remote parser.
> they might hand-edit a machine-generated one to make a change someplace in it
Ah, there's [stupendously niche use case] ;)
Seriously, though, I do agree with your point that good software should handle every edge case. I'm not arguing that.
But the case for having trailing commas does seem to be generally predicated on handwritten JSON, so I'm saying it's _unlikely_ it would be used in that way, and therefore that such failures would be rare and thus not a very grave counterargument.
Also field-names which don't contain whitespace should not need to be quoted.
Everything is a trade off. So what do we get in trade for those rather large costs?
40% bandwidth savings might be worth it. But what are the gzipped comparisons?
Human readibility is one of the most important aspects of JSON. Without that requirement you could use a binary serialization.
Unless the space after the colon is significant it seems we have to just "know" that int introduces a type definition instead of a structure.
Also
> Schema Details JSON doesn't have built-in schema support!
seems a little disingenuous. JSON provides a name for each type of value, so there is mostly no need for the schema when viewing the data. There is a JSON Schema definition.
>If you look closely, this JSON document mixes the data employees with other non-data keys (headers) such as count, currentPage, and pageSize in the same response.
But they don't explain at all how Changing the data format fixes the underlying issue of mixed concerns in one data object.
Addresses are so varied in implementation and meaning that it’s frankly ridiculous.