What would a date type look like? An ISO-8601 string is probably as good as it'll get, else you end up with an object containing loads of additional information like timezones, offsets, etc. There's also https://json-schema.org/understanding-json-schema/reference/... for a formalized date format in JSON.
The looseness seems to be aimed at human authors who want to write JSON as if it were JS, so mainly package.json. But, json is not ideal for configuration for multiple reasons.
If you wanted JSON to parse into actual date objects, you could always do:
{
thisIsADate: new Date('1995-12-17T03:24:00Z')
}
It would be simple enough for parsers in other languages to parse this into their own date objects. Much simpler than using regexs to find ISO strings.
I think allowing a whitelisted set of constructors (so it's extensible, but with some standard ones) as types in a future JSON-like could be good. e.g.
const serialised = NuJSON.stringify(data)
const result = NuJSON.parse(serialised, allowedConstructors, fallbackConstructor)
And if allowedConstructors wasn't present, it'd default to including all the standard JS ones (Map,Set,Date,Symbol,ArrayBuffer etc). The fallback constructor would be used if there were data types in the data that didn't have matching allowed constructors. By default it would through an error.
Of course, my ideal NuJSON would also include bigints, multiline strings, comments, bare keys (i.e. without having to wrap them in quotes).
Stretch nuJSON would also include the ability to serialise circular graph structures, perhaps through the use of a Reference([up, up, "aSymbol"]) data type.
I know this is how Javascript is done nowadays but using constructors for maps and arrays just seems like unnecessary syntax to me. And if you're going to initialize an array buffer too, you might as well just not use JSON at all and send plain Javascript code over the wire.
I think all JSON needs in this regard is type hinting. Let everything else be handled by the application:
{
birthdayParty: Date 1632817436514,
sounds: String? { // zomgwtfbbq comments too
cow:"moo",
fox: null
},
possibilities: ["she loves me", "she loves me not", 42],
aSymbol: Symbol "tag",
aBuffer: Uint8 [104, 101, 108, 108, 111]
}
I think the main difference between my suggestion and yours is that mine has more brackets. I think type hinting and 'constructors' are basically the same thing, but I like the types to be extensible, so even in your version, I'd make the type hints things that could be provided into the NuJSON.parse function.
Obviously, in your version my approach would be
sounds: Map {
cow: "moo",
fox: null
},
possibilities: Set ["she loves me", "she loves me not", 42]
I suppose that the idea could be that type hints are entirely ignoreable, and if you strip them the file just becomes normal JSON?
I suppose my point was that you don't have to specify types for maps and sets at all (which I didn't do in my example,) the existing syntax already does that. [] is already a set, {} is already a map. Type hints would just be used for the values.
Sending type definitions seems like a good idea. Although at that point it might as well not even be called any kind of JSON.
I don't agree that you don't have to specify types for maps and sets. For example
{true: "yes", false: "no"}
Is not valid JSON, but
Map [[true, "yes"], [false, "no"]]
should be valid 'NuJSON'. Maps and Sets are semantically different to Objects and Arrays, and they are Javascript standard objects that should ideally be supported by the default serialisation. The other alternative of course is to force the deserialisation code to fix them up afterwards, but that is painful.
I want to be able to stringify records with Sets, Maps, Arrays and Objects in, and have that work, even if they have behavior not supported by objects and arrays, and I don't think the user should have to fix up the output after parsing of JSON containing standard javascript objects.
You wouldn't say that mammals write C all the time, would you? Front end development would be so much better if you could just spew json at people in a table.
>You wouldn't say that mammals write C all the time, would you?
Yes. Mammals do write C all the time, because humans are mammals. And C is meant to be human readable and writable.
> Front end development would be so much better if you could just spew json at people in a table.
That's literally what a JSON API is, and there are tons of them. And for each of them, a human had to be able to read and comprehend JSON responses and write JSON requests in code. There's a reason JSON is pretty-printed in the debug consoles of browsers.
To say nothing of all of the package definitions or framework configs that are written (by hand) in JSON.
The fact that most JSON is machine generated at present has nothing to do with its original design intent. Most Javascript, CSS and HTML are machine generated nowadays as well, but they were meant to be written by a person using a text editor.
I've worked on 4+ years of technology that was entirely based on humans hand-tweaking, editing, and updating JSON configs. My text editors validate it quickly, and know how to fold, sort, prettify. Based on my experience (and editing it all day long) It's a very human readable and editable configuration.
Useful when you need to input already double-quoted value.
But it's a matter of taste still and nothing that we couldn't do before. Such changes will lead to meetings where programmers discuss JSON5 quoting choices (yes, it will be a thing).
It brings it closer to JavaScript behavior I guess.
I guess it's nice to have ability to copy random js object declaration to actual json without editing
I like it, most of the additions seem more like filling gaps in the original version. And the lack of trailing commas & comments have always annoyed me a bit.
But as this is a library I'm guessing NPM doesn't support it for the package.json?
Is there an official standard body behind this (and json 6 mentioned in another comment), or is it a kind of loose / informal standard left up to libraries to implement?
I mean comments and trailing commas will make JSON a bit more human friendly when used in e.g. configuration (package.json), but as a pure data exchange format it doesn't add much. Mind you, there's better data exchange formats than json, like XML (yes I said it) which can be transferred via a binary protocol (https://www.w3.org/TR/exi/) instead of the, when you think about it, awkward and inefficient data -> text format -> compression -> transfer -> decompress -> text format -> parse cycle of both 'traditional' XML and JSON both.
I wouldn't be averse to the idea of browser manufacturers or standards bodies to come up with a more efficient data transfer method that can be human-read with the right tooling like browser development tools, but which is fully compressed binary and efficient on the line. I feel like tools like GRPC are a bit too far out there.
Not sure human-readability was a major consideration of JSON really, but maybe.
One of the major considerations that many seem to miss today though was simplicity of parsing: it's a much more strict syntax than actual JS so there's a lot less ambiguity in parsing it. E.g. there's no need to have layered logic for recognizing unquoted identifiers, no need to check for multiple quote types on IDs & strings, consistent comma-ing etc. In general giving the author less options will make your parser simpler.
This is notably the exact opposite of YAML's approach.
JSON5 is nowhere near as bad as YAML, but it does seem to leaving behind some of the raison d'être of JSON. Like, I hate double-quoting in JS/JSON (personal bike-shed colour preference), but surely variable quoting characters is the worst of both worlds.
Agreed. Every single feature here complicates parsing, and the only two that are worth it IMO are comments and multi-line strings (though I'm sure others disagree on which ones are most important, which is how we end up with something like this)
Comments maybe. Multi-line strings though I don't really see as very valuable.
Lack of multi-line strings in JS was a very large, but infrequent pain for years before it was added. That infrequency tends toward zero for JSON: most of it is generated, and even when it is hand-authored, long strings are very much an exception in those cases (in my experience).
a) Short series' of shell commands in package.json
b) Longform content
I disagree that JSON is rarely hand-authored. You could argue it isn't the best medium for these usecases, but it definitely gets used for them either way
JSON is often hand-authored as small config formats (in which your use-case (a) is relevant*), but I have never seen it hand-authored for readable "content".
Firstly longform content is often rich-text; if that's hand-authored in something that's not a wysiwyg UI, it's typically a dedicated separate file (.html or .md) which may-or-may not be later programmatically compiled to JSON.
For longform content that's not rich-text, and doesn't warrant a separate dedicated .txt file: how long is it? Do you really need multiline strings?
The only real use-case I've seen is i18n string map files, where some translations can contain a newline or two, but these are an exception; the majority of strings in those files are short.
----
* On use-case (a), this is a personal preference but I tend to prefer referencing single independent executables in these use-cases, to the modern trend of having long unreadable bash one-liners in config files.
> Is there an official standard body behind this (and json 6 mentioned in another comment), or is it a kind of loose / informal standard left up to libraries to implement?
While I think this would make JSON more fun to edit by-hand, I don’t feel the breaking changes across the stack, and mixed adoption, are worth the benefits.
json-ld files (json line delimited, means one json object per line) are used extensively in data processing.
A lot of processing of these json-ld files uses normal builtins instead of special libraries eg in python doing a for line in or a stream map and the standard json lib etc.
From a glance at this spec, it seems that where an object ends is no longer something that the normal file processing most languages possess work. Which is a problem.
I’d be surprised if a extended-json ever got traction because everyone already has code that uses the lowest common denominator json parser and line splitting built into whatever language and libraries they are using and those they interchange with already use.
Having said that, the pain points I wrestle with are lack of date types and how to safely pass large integers and currencies etc. Things not addressed in the json5 spec?
I call it JSONL but yeah it's super useful. The ability to split and join using standard Unix tools, or just pipe through GNU parallel for automatic speedup without any boilerplate code
What's the point? I cringe every time a project uses both double quote and single quote, a mix of semicolon and no semicolon and a mix of tabs and space, ....
Which one to pick isn't relevant as soon as the code base is consistent
I disagree with the OP, not the standard. The standard says you can't use backticks for quoting strings, so it's still "up to them to decide which one you should go with".
My only disagreement is that you imply that this is no better than plain old JSON – I think it's positively _worse_. It makes it difficult to remember which of the two quote marks to escape, without checking the beginning of the string to see what quote marks were there (and this difficulty exists not only for humans but for machines, which introduces more complexity in parsers).
The benefit of JSON is that it's simple, opinionated, and pretty. Putting bells and whistles on it ruins the very ethos of JSON. If you want JSON++, use XML or Jsonnet or whatever.
An annoying half-solution that breaks down as soon as you encounter content with both. Better imho to have one way and become good at it. I think I might even prefer trailing commas to be mandatory.
That's exactly my point: 99% solutions are the worst. Every 99% solution will eventually find someone relying on it, willingly or not. And it's always someone else cleaning up the mess (or failing to find the cause).
They don’t serve one. Clojure uses only white space for distinguishing items in lists and hashmaps and after a short period of “this is weird” you embrace it and wonder why you ever needed commas at all
Checksums, basically, drawing a line between random bytes and structured JSON in a way that comes at zero cost for a human mind (arguably less than zero). We might as well leave out all closing tokens at the end of the stream, in a way those are completely redundant but still worth it for the warm, fuzzy feeling of knowing that you haven't just read a truncated stream.
It neatly sidesteps the whole "cat-and-mouse" issue.
Pair this with the fact that it only has the C++ style // comments, it means that every line can be tokenized independently. Not that that particularly matters for programmers, but I know Vim sometimes gets confused about where Python's strings end and it turns the syntax highlighting to garbage.
I would prefer PostgreSQL-style arbitrary quotes, but with nesting. Something like:
// the object key here is "weird object key '\""
{%{weird object key '":}%:"see how it works?"}
// ditto
{%""{weird object key '":}""%:"see how it works?"}
I'd be interested to see a counter example where a standalone backtick is part of normal non-computing related text. I really don't find your comment helpful. The strings where I see single and double quotes being used most frequently is with normal, yes, English text. The backtick is useful there, and that's all I wanted to say. French doesn't really have the same issue, since guillemets (« and ») are often used in place of double quotes, for example.
According to this Stack Exchange answer[0], the backtick is strictly for use in computing. I understand that it represents the grave accent, but that isn't of concern as a string delimiter, because once it is applied to a letter, for example è, it has its own unique representation.
I think the point was that it doesn't make sense to expect that strings in a format like JSON or a programming language will only be used for non-computing related plain text.
How about sticking with JavaScript, as long as that's what the J and the S in JSON stand for. And as we all know, JavaScript supports both single and double quotes.
Or they could pull a YAML, and retroactively redefine what JSON stands for, the way YAML originally stood for "Yet Another Markup Language" until some party pooper pointed out to their great surprise that YAML was not actually a document markup language, but a data serialization language, so they redefined it to mean "YAML Ain't Markup Language" -- the opposite of what it originally meant.
… could’ve at least disallowed for one of the most batshit crazy aspects of JSON, the legal but undefined behaviour of duplicate keys in objects.
> An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
This can be solved if you base JSON5 on I-JSON (RFC 7493) instead of JSON (RFC 8259). Many (but not all) JSON implementations actually expect I-JSON, at least for object keys.
Having a defined, predictable behaviour in JSON5 doesn't seem like it would break backwards compatibility with JSON's undefined, unpredictable behaviour. I don't think someone with an existing JSON document would complain that a JSON5 parser behaved "too predictably" when interpreting their document.
This attitude willfully ignores real-world constraints. There are plenty of "shoulds", but at the end of the day, changing behavior is changing behavior. Maybe it's worth it, maybe it isn't. But it's never free.
I've seen some of the endless flamewars about removing that.
The problem is ultimately that you kinda can't remove duplicate keys. You can only explain the ways in which handling them varies. I'm glossing over a lot of stuff though and I'll understand your skepticism.
I know it’s not trendy but consider XML and schema at this point. XML allows all this to be expressed and more and schema allows the creation of integration contracts that can be validated at both ends. Tooling and libraries are mature.
I want to love XML schemas, but to call them mature is a stretch. There's a bunch of important features hidden behind XSD 1.1, which is almost 10 years old by now, yet most software that validates XML on schemas does not support that yet.
The XML Schema standard is terrible, and its mistakes and weaknesses were what drove James Clark and Murata Makoto to develop RELAX NG (REgular LAnguage for XML Next Generation).
RELAX NG has a clearly focused sound mathematical underpinning: regular expressions applied to trees, while XML Schema is an ad-hoc hot mess designed by committee.
Any decent JSON schema language should take its cues and learn from the design of RELAX NG, and not repeat the mistakes of XML Schema.
Makoto Murata raised some critical points about XML Schema, which was beyond repair, so they both invented new regexp-based schema languages to address those problems, and combined their respective work into Relax/NG:
>Some people, including Murata and James Clark, had critical attitudes toward XML Schema. XML Schema is a modern XML schema language designed by W3C XML Schema Working Group. W3C intended XML Schema to supersede traditional DTD (Document Type Definition). XML Schema supports so many features that its specification is large and complex. Murata, James Clark and those who criticised XML Schema, pointed out the following:
>It is difficult to implement all features of XML Schema.
>It is difficult for engineers to read and write XML Schema definitions.
>It does not permit nondeterministic content models.
James Clark compared Relax NG to XML Schema and its predecessor, SGML Document Type Definitions, in his paper, "The Design of RELAX NG":
>James Clark used Haskell to design and implement an algorithm for validating Relax NG XML schemas (he co-designed Relax NG, and designed its predecessor TREX), to work the ideas out before re-implementing it in (many many more lines of tedious brittle) Java (JING). Haskel works wonderfully as a design and standard definition language, that way. [...]
A Triumph of Simplicity: James Clark on Markup Languages and XML
>If you peek under the hood of high-profile open-source projects such as Mozilla, Apache, Perl, and Python, you'll find a little program called "expat" handling the XML parsing. If you've ever used the man command on your GNU/Linux distribution, then you've also used groff, the GNU version of the UNIX text formatting application, troff. If you've ever done any work with SGML, from generating documentation from DocBook to building your own SGML applications, you've undoubtedly come acros...
James Clark argues that it's useful to have multiple specialized schema definition languages, each for their own purpose, so putting one reference to one "official" schema in one "official" language inside the document doesn't make sense.
For example, you might want to perform one kind of quick simple input sanity checking validation that ensures the input won't crash your processor, to perform in real time on millions of XML documents you receive every minute.
Then you might also want another more elaborate schema with an anal retentive formal specification, including text syntax checking, data type validation, cross-reference checking, constraints between values, and documentation, for creating and editing the same document in a schema-driven editor.
It's also a bad idea to interpret the URI of a schema as the URL from which you can download the schema definition. That introduce security and privacy and interoperability problems.
Internet Explorer used to try to download the schema definition every time you opened an XML document, which lets whoever controls that web site know you're looking at one of their documents, and what its url is in the referer field. It's like XML's answer to the 1x1 transparent GIF tracking cookie!
Unlike DTDs and XML Schema, RELAX NG doesn't presume that it's the only schema language in the world that you will ever need to use: there are other useful languages like Schematron, which do things that RELEX NG was designed not to do on purpose.
>Schematron can be useful in conjunction with many grammar-based structure-validation languages: DTDs, XML Schemas, RELAX, TREX, etc. Indeed, Schematron is part of an ISO standard (DSDL: Document Schema Description Languages) designed to allow multiple, well-focussed XML validation languages to work together. You can even embed a Schematron schema inside an XML Schema <appinfo> element or inside a RELAX NG schema!
>One part of the vision underlying RELAX NG is that validation should not be monolithic: it is not necessary or desirable to have one schema language that can handle every possible kind of validation you might want to do; it is better instead to have multiple specialized languages, each of which does one kind validation, really well. Consistent with this vision, RELAX NG provides only grammar-based validation. There's no implicit claim that other kinds of validation aren't useful and important.
>One kind of validation that is clearly useful and important and that can't be done by grammars is checking of cross-references. One possibility is to use Schematron for this. The designers of RELAX NG anticipated that there would be a little schema language specialized to this, which would be created as part of the ISO DSDL effort (as part 6); this wouldn't be a million miles from the kind of thing that XSD provides with xs:key/xs:unique/xs:keyref. Unfortunately this hasn't happened yet.
Another nice thing about RELAX NG is that not only does it have an XML format, but it also has a nice clean, much simpler to read and write, "compact syntax", which is equivalent to the XML format, and can be translated back and forth without losing any information.
It's not all goofy and arbitrary and complex line noise like XML DTDs, and it's not all verbose and clumsy namespaced XML tags and attributes like XML Scheme documents.
It even supports inheritance, and grammar as well as element level annotations (useful for schema and meta processing tools and editors).
I know this is a stupid objection, but I suspect one of the main impedances to takeup is going to be its stupid name. It's trivial I know, bikeshedding I know, but come on, "RELAX NG"?
Like, how do you pronounce that? I want to say 'Relax-ung', but it sounds odd, like you're trying to say "relaxing" but failing; or maybe 'Relax Nig', but that sounds like something you'd see in an HN thread on Timnit Gebru's firing. It doesn't feel like the naming was considered from a normal human point of view, as opposed to from the golden mountain of abstraction.
RELAX NG was a combination of Makoto Murata's work on RELAX, an acronym for "REgular LAnguage description for Xml", and James Clark's work on "TREX", an acronym for "Tree Regular EXpressions".
They could have pulled a slick smooth move and called it "TREX-LAX", for "Tree Regular EXpressions for LAnguages of Xml", which rolls off the tongue more mellifluously!
TREX-LAX: It makes your XML nice and regular, and keeps you shit moving smoothly.
XML is a terrible data interchange format because it embeds its schema in its data. That makes it sometimes an order of magnitude larger, and it takes longer to parse.
If you want to drop JSON altogether, something like Protobufs would be better. But JSON + JSONSchema has all the benefits of XML and none of the drawbacks.
XML DTDs were inherited from SGML. They can also be defined externally, but with internal links to them embedded in the document.
XML Schema were supposed to fix all the problems of DTDs, but they don't support unordered content, and have a host of their own problems. XML Schema are typically external XML documents referenced by an xsi:schemaLocation attribute, but apparently you can embed them too. But I don't know if that's a common practice, since it's a pretty bad idea for a lot of reasons:
You raise a good point that it's a bad idea to embed schemas in documents, or even references to schemas in documents, because, as James Clark points out, magic schema attributes in documents (like xsi:schemaLocation) pre-suppose that there can be only one way of validating a document, they create security and interoperability problems, and it infects documents with the namespace of the grammar that you're using to validate them, when documents shouldn't depended on such knowledge, and be forced to include hot messes of namespace attributes like:
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting: [...]
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
Here's more of what he has to say about JSON in his blog post "XML vs the Web":
1. It's super verbose. Everybody hates writing it. And yes, people do write JSON by hand.
2. It has a weird and confusing data model with both attributes and child elements. Do you do <foo a="1"/> or <foo><a>1</a></foo>? In many many cases it's ambiguous and you end up with a weird mix, whereas in JSON it's obvious and easy.
I've heard horror stories from a friend who had to fix such code introduced by a contractor. Management didn't want to throw any of it awayz because they paid a lot of money to the contractor. Of course broke in a hard to debug fashion.
It will work only as long as the object(s) interpreted by repr are serialzable by JSON, which is not a sound approach, unless you write a any-type serializer.
The syntax for line breaks make it incompatible with JavaScript.
Probably not terrible because doing an eval of untrusted json would be a bad idea but they could have used back ticks which would have these useful properties:
Sure, for APIs comments are bad. However, for configuration things are so much better when you can document or temporarily remove things, without having hacks which pollute the data.
Trailing commas are also nice to have - although we have linters for that, unfortunately some people think they're too good to need a linter.
I like that it has a new name. I've seen too many stupid libraries support their own superset while still calling it JSON (looking at you, Python and .NET ecosystem!)
It's a standard. If we change it, it needs a new name..
However, I am worried that this new standard is harder for languages and ecosystems to support consistently, because it requires a much more involved parser.
Then again, if YAML can do it.. (and that must be a parser created in the depths of hell).
"JSON (JavaScript Object Notation) is a lightweight data-interchange format."
So says JSON.org for as long as anyone can remember.
Why is anyone caring about using single or double quotes, or wanting trailing commas? It solves no problems. It seems to me the only answer is it's for manually created JSON. That is, all of this is for people whose only exposure to JSON is doing some config in vscode. It adds nothing else.
Keep going, next sentence from JSON.org: "It is easy for humans to read and write. It is easy for machines to parse and generate. "
Non-quoted keys is easier to read and write by humans, trailing commas are easier for a machine to write, single quotes can make it easier for a human to read or write. Comments help humans read and write.
I remember when it was javascript object notation and that stuff was allowed because thats what javascript allows and felt simple and effective for both human and machine, then it became strict json, what a step backwards.
Regardless of how good this is or not, every time it pops up I just wonder if anybody actually gives a crap or not.
JSON is one of those formats that's "good enough" even if it's stinky and it's also universally accepted that a small iteration on it with small conveniences doesn't make that much sense imho.
And XML is a very battle-tested alternative that supports every conceivable edge case and arbitrary depths of strictness and validation. There was soooo much energy poured in to making XML the One Format To Rule Them All. It has so many advanced features like strict definitions, includes syntax, support for mixed content, entities and references, comment syntax. But it turns out that 99% of users never needed any of that and just wanted nested key-value pairs.
A serialization format is more than just how things are described on the wire. It's also about how both sides are expected to receive and transmit it. Naive processing of JSON in certain situations can run into sharp edges like this:
Serialization formats are a part of Protocols, which essentially form the software equivalent of a spoken language between two people. Like words, the bytes can have a general syntax and expected semantic, but still be open to problematic interpretation. Like the conceptual difference between a language like English and Ithkuil. (http://www.newyorker.com/magazine/2012/12/24/utopian-for-beg...)
Like Ithkuil, some formats can specify "a little more" in a way that tries to prescribe semantic meaning and syntax such that implementing according to spec limits the opportunities for issues like this to happen.
JSON is "good enough" for a textual, mostly human readable data interchange format. But as a configuration format, for example the package.json file for node/npm/yarn, it is most definitely not "good enough". For two reasons: no comments and no multiline strings. The latter might not be important for some types of configuration, but it is for things that need something like a description that is more than a single sentence.
> But as a configuration format, for example the package.json file for node/npm/yarn, it is most definitely not "good enough".
Then perhaps don't use it for a use case it was not designed to address? I understand that JSON is a very handy hammer to have, but not all things are nails.
I mean, that’s all very well, but it’s literally one of only two formats with native support in the browser. So there’s plenty of times you don’t really have any choice other than to hammer that screw in.
It is just that people keep insisting on using JSON for human-maintained configuration files, and JSON sucks for that: no comments, trailing commas, or multi-line strings.
Since JSON5 is trivially pre-processed into JSON, it tends to be useful as a tack-on to such projects to let your startup scripts generate the real file given the useful on without having unrelated syntax.
156 comments
[ 3.7 ms ] story [ 234 ms ] thread[0] https://hjson.github.io/
The looseness seems to be aimed at human authors who want to write JSON as if it were JS, so mainly package.json. But, json is not ideal for configuration for multiple reasons.
Of course, my ideal NuJSON would also include bigints, multiline strings, comments, bare keys (i.e. without having to wrap them in quotes).
Stretch nuJSON would also include the ability to serialise circular graph structures, perhaps through the use of a Reference([up, up, "aSymbol"]) data type.
I think all JSON needs in this regard is type hinting. Let everything else be handled by the application:
Obviously, in your version my approach would be
I suppose that the idea could be that type hints are entirely ignoreable, and if you strip them the file just becomes normal JSON?Sending type definitions seems like a good idea. Although at that point it might as well not even be called any kind of JSON.
I want to be able to stringify records with Sets, Maps, Arrays and Objects in, and have that work, even if they have behavior not supported by objects and arrays, and I don't think the user should have to fix up the output after parsing of JSON containing standard javascript objects.
is this any better?
in many languages single quotes are used as apostrophes and are very very common, much more common than double quotes."I'm \"really\" angry, don't do it again!"
Yes. Mammals do write C all the time, because humans are mammals. And C is meant to be human readable and writable.
> Front end development would be so much better if you could just spew json at people in a table.
That's literally what a JSON API is, and there are tons of them. And for each of them, a human had to be able to read and comprehend JSON responses and write JSON requests in code. There's a reason JSON is pretty-printed in the debug consoles of browsers.
To say nothing of all of the package definitions or framework configs that are written (by hand) in JSON.
The fact that most JSON is machine generated at present has nothing to do with its original design intent. Most Javascript, CSS and HTML are machine generated nowadays as well, but they were meant to be written by a person using a text editor.
But it's a matter of taste still and nothing that we couldn't do before. Such changes will lead to meetings where programmers discuss JSON5 quoting choices (yes, it will be a thing).
Wait, that wasn't possible before?
I like it, most of the additions seem more like filling gaps in the original version. And the lack of trailing commas & comments have always annoyed me a bit.
But as this is a library I'm guessing NPM doesn't support it for the package.json?
I mean comments and trailing commas will make JSON a bit more human friendly when used in e.g. configuration (package.json), but as a pure data exchange format it doesn't add much. Mind you, there's better data exchange formats than json, like XML (yes I said it) which can be transferred via a binary protocol (https://www.w3.org/TR/exi/) instead of the, when you think about it, awkward and inefficient data -> text format -> compression -> transfer -> decompress -> text format -> parse cycle of both 'traditional' XML and JSON both.
I wouldn't be averse to the idea of browser manufacturers or standards bodies to come up with a more efficient data transfer method that can be human-read with the right tooling like browser development tools, but which is fully compressed binary and efficient on the line. I feel like tools like GRPC are a bit too far out there.
One of the major considerations that many seem to miss today though was simplicity of parsing: it's a much more strict syntax than actual JS so there's a lot less ambiguity in parsing it. E.g. there's no need to have layered logic for recognizing unquoted identifiers, no need to check for multiple quote types on IDs & strings, consistent comma-ing etc. In general giving the author less options will make your parser simpler.
This is notably the exact opposite of YAML's approach.
JSON5 is nowhere near as bad as YAML, but it does seem to leaving behind some of the raison d'être of JSON. Like, I hate double-quoting in JS/JSON (personal bike-shed colour preference), but surely variable quoting characters is the worst of both worlds.
Lack of multi-line strings in JS was a very large, but infrequent pain for years before it was added. That infrequency tends toward zero for JSON: most of it is generated, and even when it is hand-authored, long strings are very much an exception in those cases (in my experience).
a) Short series' of shell commands in package.json
b) Longform content
I disagree that JSON is rarely hand-authored. You could argue it isn't the best medium for these usecases, but it definitely gets used for them either way
Firstly longform content is often rich-text; if that's hand-authored in something that's not a wysiwyg UI, it's typically a dedicated separate file (.html or .md) which may-or-may not be later programmatically compiled to JSON.
For longform content that's not rich-text, and doesn't warrant a separate dedicated .txt file: how long is it? Do you really need multiline strings?
The only real use-case I've seen is i18n string map files, where some translations can contain a newline or two, but these are an exception; the majority of strings in those files are short.
----
* On use-case (a), this is a personal preference but I tend to prefer referencing single independent executables in these use-cases, to the modern trend of having long unreadable bash one-liners in config files.
There's a spec: https://spec.json5.org/
A lot of processing of these json-ld files uses normal builtins instead of special libraries eg in python doing a for line in or a stream map and the standard json lib etc.
From a glance at this spec, it seems that where an object ends is no longer something that the normal file processing most languages possess work. Which is a problem.
I’d be surprised if a extended-json ever got traction because everyone already has code that uses the lowest common denominator json parser and line splitting built into whatever language and libraries they are using and those they interchange with already use.
Having said that, the pain points I wrestle with are lack of date types and how to safely pass large integers and currencies etc. Things not addressed in the json5 spec?
What you describe seems to be called line-delimited JSON, LDJSON, NDJSON or JSONL.
https://en.wikipedia.org/wiki/JSON_streaming
What's the point? I cringe every time a project uses both double quote and single quote, a mix of semicolon and no semicolon and a mix of tabs and space, .... Which one to pick isn't relevant as soon as the code base is consistent
E.g., how would you put the following sentence in a string:
You can use both " and ' to delimit a string.
"You can use both \" and ' to delimit a string"
The benefit of JSON is that it's simple, opinionated, and pretty. Putting bells and whistles on it ruins the very ethos of JSON. If you want JSON++, use XML or Jsonnet or whatever.
The 99% case is a pretty string. The 1% case is a string with escaped delimiters.
The most practical solution for this problem I saw was in F#. You can use tripple-quoted strings.
https://docs.microsoft.com/en-us/dotnet/fsharp/language-refe...Pair this with the fact that it only has the C++ style // comments, it means that every line can be tokenized independently. Not that that particularly matters for programmers, but I know Vim sometimes gets confused about where Python's strings end and it turns the syntax highlighting to garbage.
According to this Stack Exchange answer[0], the backtick is strictly for use in computing. I understand that it represents the grave accent, but that isn't of concern as a string delimiter, because once it is applied to a letter, for example è, it has its own unique representation.
0: https://english.stackexchange.com/questions/249406/are-backt...
Or they could pull a YAML, and retroactively redefine what JSON stands for, the way YAML originally stood for "Yet Another Markup Language" until some party pooper pointed out to their great surprise that YAML was not actually a document markup language, but a data serialization language, so they redefined it to mean "YAML Ain't Markup Language" -- the opposite of what it originally meant.
https://en.wikipedia.org/wiki/YAML#History_and_name
https://en.wikipedia.org/wiki/Markup_language
Pro Tip: When naming you file format, look up what the words in your acronym mean, first.
Some code bases use single quotes, and it's annoying to copy/paste an object from that code into a JSON objdct and have to change the quotes.
> An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
I've seen some of the endless flamewars about removing that.
The problem is ultimately that you kinda can't remove duplicate keys. You can only explain the ways in which handling them varies. I'm glossing over a lot of stuff though and I'll understand your skepticism.
It hasen't changed since then and it will never need to change ever in the history of mankind.
Problem solved!
Edit: Maybe this solves another problem though, but I doubt it!
https://en.wikipedia.org/wiki/RELAX_NG
RELAX NG has a clearly focused sound mathematical underpinning: regular expressions applied to trees, while XML Schema is an ad-hoc hot mess designed by committee.
https://en.wikipedia.org/wiki/XML_schema#RELAX_NG
Any decent JSON schema language should take its cues and learn from the design of RELAX NG, and not repeat the mistakes of XML Schema.
Makoto Murata raised some critical points about XML Schema, which was beyond repair, so they both invented new regexp-based schema languages to address those problems, and combined their respective work into Relax/NG:
https://en.wikipedia.org/wiki/Makoto_Murata#RELAX_and_RELAX_...
>Some people, including Murata and James Clark, had critical attitudes toward XML Schema. XML Schema is a modern XML schema language designed by W3C XML Schema Working Group. W3C intended XML Schema to supersede traditional DTD (Document Type Definition). XML Schema supports so many features that its specification is large and complex. Murata, James Clark and those who criticised XML Schema, pointed out the following:
>It is difficult to implement all features of XML Schema.
>It is difficult for engineers to read and write XML Schema definitions.
>It does not permit nondeterministic content models.
James Clark compared Relax NG to XML Schema and its predecessor, SGML Document Type Definitions, in his paper, "The Design of RELAX NG":
https://relaxng.org/jclark/design.html
James Clark has a huge amount of experience designing and implementing SGML and XML standards:
https://en.wikipedia.org/wiki/James_Clark_(programmer)
I've written about XML Schema, Relax NG, and James Clark eariler:
https://news.ycombinator.com/item?id=26122033
>Some of the most incredibly awfully bad XML DSLs are official standards, themselves. COUGH XMLSchema COUGH: [...]
https://news.ycombinator.com/item?id=22756875
>James Clark used Haskell to design and implement an algorithm for validating Relax NG XML schemas (he co-designed Relax NG, and designed its predecessor TREX), to work the ideas out before re-implementing it in (many many more lines of tedious brittle) Java (JING). Haskel works wonderfully as a design and standard definition language, that way. [...]
A Triumph of Simplicity: James Clark on Markup Languages and XML
https://www.drdobbs.com/a-triumph-of-simplicity-james-clark-...
>If you peek under the hood of high-profile open-source projects such as Mozilla, Apache, Perl, and Python, you'll find a little program called "expat" handling the XML parsing. If you've ever used the man command on your GNU/Linux distribution, then you've also used groff, the GNU version of the UNIX text formatting application, troff. If you've ever done any work with SGML, from generating documentation from DocBook to building your own SGML applications, you've undoubtedly come acros...
For example, you might want to perform one kind of quick simple input sanity checking validation that ensures the input won't crash your processor, to perform in real time on millions of XML documents you receive every minute.
Then you might also want another more elaborate schema with an anal retentive formal specification, including text syntax checking, data type validation, cross-reference checking, constraints between values, and documentation, for creating and editing the same document in a schema-driven editor.
https://github.com/michmech/xonomy
It's also a bad idea to interpret the URI of a schema as the URL from which you can download the schema definition. That introduce security and privacy and interoperability problems.
Internet Explorer used to try to download the schema definition every time you opened an XML document, which lets whoever controls that web site know you're looking at one of their documents, and what its url is in the referer field. It's like XML's answer to the 1x1 transparent GIF tracking cookie!
Unlike DTDs and XML Schema, RELAX NG doesn't presume that it's the only schema language in the world that you will ever need to use: there are other useful languages like Schematron, which do things that RELEX NG was designed not to do on purpose.
https://www.schematron.com/
(There's a great Ven diagram comparing XSD, DTD, RELAX NG and Schematron's scope on that page.)
https://www.schematron.com/document/2755.html
>Schematron can be useful in conjunction with many grammar-based structure-validation languages: DTDs, XML Schemas, RELAX, TREX, etc. Indeed, Schematron is part of an ISO standard (DSDL: Document Schema Description Languages) designed to allow multiple, well-focussed XML validation languages to work together. You can even embed a Schematron schema inside an XML Schema <appinfo> element or inside a RELAX NG schema!
James Clark on RELAX NG and xml:id:
https://blog.jclark.com/2009/01/relax-ng-and-xmlid.html
>RELAX NG and xml:id
>One part of the vision underlying RELAX NG is that validation should not be monolithic: it is not necessary or desirable to have one schema language that can handle every possible kind of validation you might want to do; it is better instead to have multiple specialized languages, each of which does one kind validation, really well. Consistent with this vision, RELAX NG provides only grammar-based validation. There's no implicit claim that other kinds of validation aren't useful and important.
>One kind of validation that is clearly useful and important and that can't be done by grammars is checking of cross-references. One possibility is to use Schematron for this. The designers of RELAX NG anticipated that there would be a little schema language specialized to this, which would be created as part of the ISO DSDL effort (as part 6); this wouldn't be a million miles from the kind of thing that XSD provides with xs:key/xs:unique/xs:keyref. Unfortunately this hasn't happened yet.
[...]
It's not all goofy and arbitrary and complex line noise like XML DTDs, and it's not all verbose and clumsy namespaced XML tags and attributes like XML Scheme documents.
It even supports inheritance, and grammar as well as element level annotations (useful for schema and meta processing tools and editors).
RELAX NG's Compact Syntax:
https://www.xml.com/pub/a/2002/06/19/rng-compact.html
RELAX NG Compact Syntax: Committee Specification 21 November 2002:
https://www.oasis-open.org/committees/relax-ng/compact-20021...
RELAX NG Compact Syntax Tutorial:
https://relaxng.org/compact-tutorial-20030326.html
Like, how do you pronounce that? I want to say 'Relax-ung', but it sounds odd, like you're trying to say "relaxing" but failing; or maybe 'Relax Nig', but that sounds like something you'd see in an HN thread on Timnit Gebru's firing. It doesn't feel like the naming was considered from a normal human point of view, as opposed to from the golden mountain of abstraction.
They could have pulled a slick smooth move and called it "TREX-LAX", for "Tree Regular EXpressions for LAnguages of Xml", which rolls off the tongue more mellifluously!
TREX-LAX: It makes your XML nice and regular, and keeps you shit moving smoothly.
https://www.browncafe.com/community/threads/fedex-and-ex-lax...
>FedEx and Ex-Lax battle for rights to promotional slogan
JSON-schema is a thing as well, maybe not quite as mature (or "aged"), but the tooling is fine.
If you want to drop JSON altogether, something like Protobufs would be better. But JSON + JSONSchema has all the benefits of XML and none of the drawbacks.
XML Schema were supposed to fix all the problems of DTDs, but they don't support unordered content, and have a host of their own problems. XML Schema are typically external XML documents referenced by an xsi:schemaLocation attribute, but apparently you can embed them too. But I don't know if that's a common practice, since it's a pretty bad idea for a lot of reasons:
https://www.herongyang.com/XML/XSD-Statements-Embedded-in-XM...
You raise a good point that it's a bad idea to embed schemas in documents, or even references to schemas in documents, because, as James Clark points out, magic schema attributes in documents (like xsi:schemaLocation) pre-suppose that there can be only one way of validating a document, they create security and interoperability problems, and it infects documents with the namespace of the grammar that you're using to validate them, when documents shouldn't depended on such knowledge, and be forced to include hot messes of namespace attributes like:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0... http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd">
Schema Wars: XML Schema vs. RELAX NG (1/2) - exploring XML:
https://web.archive.org/web/20190403171512/http://webreferen...
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting: [...]
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
Here's more of what he has to say about JSON in his blog post "XML vs the Web":
https://blog.jclark.com/2010/11/xml-vs-web_24.html
>[...] From this perspective, my reaction to JSON is a combination of "Yay" and "Sigh".
1. It's super verbose. Everybody hates writing it. And yes, people do write JSON by hand.
2. It has a weird and confusing data model with both attributes and child elements. Do you do <foo a="1"/> or <foo><a>1</a></foo>? In many many cases it's ambiguous and you end up with a weird mix, whereas in JSON it's obvious and easy.
Probably not terrible because doing an eval of untrusted json would be a bad idea but they could have used back ticks which would have these useful properties:
- still JavaScript
- fewer characters
Sure, for APIs comments are bad. However, for configuration things are so much better when you can document or temporarily remove things, without having hacks which pollute the data.
Trailing commas are also nice to have - although we have linters for that, unfortunately some people think they're too good to need a linter.
IMO the real value in trailing commas is that it's easier to produce simple JSON programmatically.
Ah, it's based on ECMAScript 5.1. I guess we can wait for a JSON6, then.
{ justDate: @2020-02-08 }
{ dateAndTime: @2007-03-01T13:00:00Z }
{ justTime: @@13:00:00 }
But I wouldn't ever use JSON5 over JSON for serialization, or over YAML for parsing a configuration file.
I think at this point a new json spec should have little/nothing to do with javascript.
It's a standard. If we change it, it needs a new name..
However, I am worried that this new standard is harder for languages and ecosystems to support consistently, because it requires a much more involved parser.
Then again, if YAML can do it.. (and that must be a parser created in the depths of hell).
So says JSON.org for as long as anyone can remember.
Why is anyone caring about using single or double quotes, or wanting trailing commas? It solves no problems. It seems to me the only answer is it's for manually created JSON. That is, all of this is for people whose only exposure to JSON is doing some config in vscode. It adds nothing else.
Call it JSON-CONFIG and leave well enough alone.
Non-quoted keys is easier to read and write by humans, trailing commas are easier for a machine to write, single quotes can make it easier for a human to read or write. Comments help humans read and write.
I remember when it was javascript object notation and that stuff was allowed because thats what javascript allows and felt simple and effective for both human and machine, then it became strict json, what a step backwards.
JSON is one of those formats that's "good enough" even if it's stinky and it's also universally accepted that a small iteration on it with small conveniences doesn't make that much sense imho.
You trade slightly more concrete data structures for hard incompatibility.
Maybe an internal api could leverage this in some way?
But it really isn't that hard to wrap hex, floating points, etc. in quotes and parse that server side.
Any application that's not a pet project should be using json schemas anyways, which is the layer you could put your custom datatypes into.
Every serialization format has security implications.
Some try to make this explicit and probably fail in capturing all conceivable holes. Some leave this implicit to give you "fun surprises."
https://rules.sonarsource.com/java/RSPEC-4544
Serialization formats are a part of Protocols, which essentially form the software equivalent of a spoken language between two people. Like words, the bytes can have a general syntax and expected semantic, but still be open to problematic interpretation. Like the conceptual difference between a language like English and Ithkuil. (http://www.newyorker.com/magazine/2012/12/24/utopian-for-beg...)
Like Ithkuil, some formats can specify "a little more" in a way that tries to prescribe semantic meaning and syntax such that implementing according to spec limits the opportunities for issues like this to happen.
Then perhaps don't use it for a use case it was not designed to address? I understand that JSON is a very handy hammer to have, but not all things are nails.
Since JSON5 is trivially pre-processed into JSON, it tends to be useful as a tack-on to such projects to let your startup scripts generate the real file given the useful on without having unrelated syntax.
The JSON5 Data Interchange Format (https://spec.json5.org/) - 2 Years ago - 0 Comments - https://news.ycombinator.com/item?id=21113613