The idea was/is to capture the spirit of JSON minimalism, avoiding the insanity of XML-based schema languages.
Currently, some UCSC students are working on javascript and java tools to work with schemas. We should be able to release that work in the next month or so.
There's also JSONSchema (http://json-schema.org/), which is significantly more verbose, but has optionals, string length constraints and other sort of quite useful stuff.
And there is (or, better say, was) an "Orderly JSON" language that had nice and compact human-manageable representations, but had essentially died from nobody using it.
JSchema:
{
"name": "@string",
"age": "@int"
}
Pros: is JSON, is simple to read and write. Cons: is (currently) limited to very primitive schemas.
Pros: is JSON and reasonably powerful. Seems to be a most common standard out there, with lots of libraries available. Cons: is verbose to the extent it's no fun to read or write by hand. Has some ambiguities and - as the OP article correctly says - spec is unfinished.
Orderly JSON:
object {
string name;
integer {0,} age?;
};
Pros: compact and powerful. Cons: not JSON but a custom grammar, no one uses it.
We use JSONSchema in several places. It has met our needs pretty well, particularly for dynamic form generation (using the angular-schema-form library)
As long as you don't need good interop, JSON Schema can be pretty great. (If you need interop, it's not, because none of the tools seem to implement unspecified behavior the same way.)
Even when you need interoperation, the well-defined subset of JSONSchema may do reasonably well. I think every library out there handles the essentials (the core types and basic constraints) in the exactly same manner, and the incompatibilities starts with less common things like references etc.
On a related note, Transit [1], by Hickey et all, can provide a meta scheme for data interchange that inter-operates with JavaScript. It provides a richer set of data sets than traditional schemas. You can define things as a Set or List, for example. As a result, you can express data behavior while expressing general structure.
Do you have any data on that? We found that using transit could actually increase overall throughput because it does some minimal compression/deduplication.
"If it looks like a document, use XML. If it looks like an object, use JSON. It’s that simple. The essential difference isn’t simplicity/complexity or compact/verbose or typed/text, it’s ordered-by-default or not."
I can appreciate that an order-independent representation can make schema versioning easier, but what other advantages does this provide?
Mostly for programming languages with hash tables / dictionaries.
Python dictionaries hold keys but the keys aren't in any particular order, so allowing them to come out in whatever order maps very naturally to the language.
is incredibly surprising and frustrating, and XML schema actually supports ignoring the difference but xml schema tools seem to hate using the "all" instead of the "sequence" schema definitions.
The worse offender, imho, is Microsoft's DataContractSerializer, whose default behavior is to silently ignore this ordering failure and also silently fail to deserialize any out-of-order elements.
"If it looks like a document, use XML." And the reason for that is XML originated from SGML and both were initially focused on marking up documents (where everything is ordered, just like the words in this sentence), not objects or bits of data. And that's why JSON is usually simpler and more useful for your example and similar. Whether or not the order is significant in your XML example becomes a schema-level issue (DTD, Relax, XMLSchema). Maybe you are being forced to use XML, but if not, you're describing an object, so use JSON.
Yes, but xmlschema allows for a decent platform-agnostic system for defining objects. The problem is that most tools default (or even force) to "sequence" instead of "all". With "all" you get object-ish XML behavior.
There is no well-adopted analog to xmlschema for json.
XPath, XSLT and XQuery are pretty awesome, in their special way.
Although stringly-typed XPath has the same problems as SQL, building complex queries through string concatenation. This is where JSON shines, because you usually just convert to a tree of your language's native data structures in one line and use your languages native facilities to traverse it from there.
Devs who think JSON is wonderful to work with probably think so because they're using a dynamically typed language.
The advantage of a schema is you can validate your data and move to static typing in only a few lines of code. It's for this reason that I think the best option these days is to just use something like protobufs v3, which has a stable JSON serialization and a sensible schema specification.
But even for objecty data, rather than "documents", XML doesn't have to be clumsy. Checkout the code samples here for C++
XML has its bad sides (hell, I won't ever forget hacking Perl's SOAP::Lite to make it be able to talk with some extremely enterprisey Java service), but its core isn't as bad as many try to picture it.
The really good thing about XML, that it - unlike JSON - is extensible.
And traversing both JSON and XML (without the weird parts) are no-brainers. Every (or about so) mainstream language has libraries that make either format really transparent to use.
I did some toy project in C# and haven't found handling JSON any painful. Maybe I just haven't hit the bad cases, though.
I don't exactly remember what I've used and what I wrote, but I think I had just installed some library (IIRC it was Json.NET), put some annotations on classes, and got my serialization. For deserialization I didn't wanted to write throwaway classes so I just made a JSONPath query and iterated over the result objects, looking at necessary properties. Nothing particularly different from how things are in, say, Python, except maybe for some extra type checking.
My biggest problem with using JSON, apart from the needless quotation marks on identifiers, is the lack of comments. Any human-editable format (like using .json for project config files or build files) should have comments.
I'm in favor of using YAML [0] or INI [1] formats when a file is meant to be edited by a user. YAML is close enough to JSON, but does support comments, and the structure is a bit cleaner for user editing.
If you do not care too much about exchanging data with other apps then parser/generator supporting javascript style comments (e.g. json-cpp) would work. With each JSON value it can store "commentBefore" and "commentAfter". If you need you can strip these comments any time by parse + write cycle with "collectComments" option inactive.
Adding data and calling it a comment isn't really a comment.
And I find the whole "run minifiy/strip first" to be a hilarious suggestion from Douglas Crockford. It can be rephrased as "If you want comments in JSON, for a config file say, then don't use JSON."
That, and XML requiring the tag name in the closing tag, are some of the obvious, silly, mistakes. XML in particular. Without the named closing tags, just using, say, <>, would reduce the apparent bloat and annoyance by a very large factor.
In statically typed languages, JSON is awful and clumsy compared to XML which has a standard parser and query language. There's pretty much no case where JSON is easier to traverse than XML. I'd challenge you to back up that statement.
I really liked what Atom did with CSON (CoffeeScript object notation). It feels very similar to CSS, in fact you specify a CSS selector and then a string representation of a key sequences and map it to an action.
Looking at the code on their project's page it doesn't seem to quite match what you wrote but maybe they don't have examples that cover it. I'm not quite sure what your example is supposed to do but those can be perfectly valid JSON keys and values so if it's doing anything but straight translating it then that's kinda weird.
I've also been surprised that there are no JSON editors that are both powerful and attractive. The one that comes closest in my experience is http://www.jsoneditoronline.org/ but I'd appreciate pointers to any others.
Unlike XML and UML, there's just less of a market for GUI-like editors for JSON?
Different uses I guess. I'm using JSON for configuration purposes (~100kB...4MB PABX config, but Azure seems to be more popular example) and it's read-modify-update cycle > 95% of the time.
I found one (windows based) a while back, but cannot for the life of me remember the name... really liked it. It was similar to XML Notepad from MS, but for JSON. Unfortunately the name of it was NOT "JSON Notepad" which would have made it far easier to find again. :-(
Something that would also be nice, if anyone comes up with such a beast would be the ability to edit a multi-record json file... effectively each line is a single JSON, and an LF at the end of each line/record. I've used this structure a few times. Given that JSON will typically encode special characters allowing for single-line representation by default, it's worked really well, especially combined with gzip as a stream for archiving data.
If anyone knows what that program was, would love to find it again.
I hope this would be JSONedit as it is mine ;) - http://tomeko.net/software/JSONedit/
There are also - I think two different - plugins for Notepad++, JsonViewer: https://jsonviewer.codeplex.com/, commercial: XMLSpy and JSONBuddy and one or two half-baked projects on codeproject.
A word of caution: some editors will choke on large files or, in particular, long lines.
Opening a one-line JSON file of a few MB in Emacs can be painful. The editor struggles to handle the long line and the matching-parenthesis-highlighter runs its stack machine down the whole length looking for the closing delimiter.
There are ways to avoid this, like pretty-printing the JSON across multiple lines, or forcing fundamental-mode when large files are opened, but I still haven't set these up as default.
(Emacs) That's weird. Smart editor should render only portion visible to user and number of lines should not make any difference for highlighter. I've checked Scintilla with 3MB single-line JSON and using C++ lexer (actually: "Lexer for C++, C, Java, and JavaScript") and it has no problems except for some lack of fluency when scrolling horizontally.
jdorn's (awesome) tool is a dynamic UI builder where the fields, headings, constraints are pulled from a json schema document (which is also a json document).
I believe the question was about good editors for json documents.
Yeah, so I do a lot with JSON right now, and having some kind of spec for APIs is pretty important. My biggest pain point is dates, since there's no simple way to throw some JSON at JSON.parse and get a date out. You have to know, a priori, what fields are dates and do some follow-up. There's other issues with some fields sometimes getting serialized as string representations of themselves (e.g.: "10" vs 10, or "false" vs false).
On the other hand, I did a lot with XML in the 00s, and I sure don't want to go back to dealing with that anymore. I'm sure the world has come a long way since SAX was the thing to use for XML, but that experience left me so deeply scarred that I'm willing to just write an ad hoc schema on my frontend and say, "after JSON.parse, go in and coerce these fields to be int / bool / Date, etc". Not great for interop, but what the hey.
Why we could not have # as a date prefix and suffix boggles my mind. And comments, if we want to use JSON for a config file we need to agree to allow // and /\ to temporarily comment out properties and object.
Is your idea of having # as a date prefix and suffix compatible with having json be a subset of valid javascript? Because that was one of its original selling points.
Are you thinking something like
{
"business": "Jim's Hat Shop",
"date_visited": #2016-05-03#
}
Or were you thinking
{
"business": "Jim's Hat Shop",
"date_visited": "#2016-05-03#"
}
Personally, this isn't bad - some kind of literal notation for date objects seems like a good idea (we already have octal and binary literals).
The other option ("#date#") is bad, IMO, since it's a magic string. I don't like magic strings because they blow up at unexpected times. Imagine the notice to the users: "please stop using the hashtag #racecar#, we know palindromes are cool, but it is causing our database to crash". I had similar issues with CDATA blocks in XML, actually - escaping strings to afford reasonable parsing is never fun.
Your note about octal and binary literals sort of goes back and forth compared to the idea of date literals for me. Because octal and binary literals are supported in vanilla javascript, but are not explicitly supported in JSON. However, throwing them in seems like only a minor transgression since they are native to javascript, and are effectively just another representation of a number.
On the other hand, the date literal you suggest is not part of javascript. And on that note, why not just use timestamps for dates in json?
But isn't it a huge change to make json not be valid javascript? These days no one in good conscience just uses "eval()" in a js app, so maybe it doesn't matter anymore, but still, it seems like that is suggesting a massive departure.
> There's other issues with some fields sometimes getting serialized as string representations of themselves (e.g.: "10" vs 10, or "false" vs false).
With JSON.parse()? That would be a big failure of handling standard JSON if it did that. Do you have an example where JSON.parse() doesn't correctly parse JSON in the manner you suggest? Or really in any manner? I haven't seen it fail yet and if there are known failures it would be good to know about them.
I have the failure occasionally when I'm just shoving a JSON object into a PUT request, and the on the server side (hapijs, postgres) taking part of that JSON object and shoving it into a jsonb column type. In certain circumstances, the value in the database row was "10" rather than 10, which caused issues in queries against it (like WHERE documents.extended->>agenda_id = 10).
I don't know where the issue really originated which part of transport or serialization, and I honestly don't know how common it really is - I just slapped some type coercions on the object I was putting into the jsonb field (which was necessary anyway, to protect against malicious user input) and called it a day.
So it's probably not JSON.parse at fault here.
EDIT to add: just looked over my notes, and it seems like the problem was likely originating in how I was treating option elements underneath a select in an html form. Somehow the value was getting set to a "10" instead of 10 on the form's model object, and that was getting pushed over the wire (serialized and deserialized) as a string.
This was my personal pet peeve for a while. I even half-assed a spec language called RELAX JSON that I bet Bray would like. [0]
But then I realized that what I had made up was very similar to type definitions in TypeScript!
Especially if you use `interface` and not `class`, TypeScript gives you a lot of flexilibity in specifying data structures. You get optional values, typed arrays, typed but mixed arrays, references to other named types that you define yourself, and so on. Really, it's all you need and more. The only downside is that you can spec stuff that you can't put in JSON, such as Date objects. As a schema language, that is a bit weird. But just don't spec it like that! :-D
I'll post some examples in a reply when I'm back behind a computer.
For the record, JSON Schemas are quite a bit more expressive than TypeScript interfaces, or even Scala Traits. I actually put together a big list of every JSON Schema constraint that isn't checkable at compile time in TS: https://github.com/bcherny/json-schema-to-typescript#not-exp....
Fair enough on the constraints. One of the design goals I had at the time was that I didn't want stuff like that to be expressible. Basically, it felt over the top to me - you might as well just allow writing predicates in plain JS then. Even more expressive!
My primary goal was a spec language for humans to read and write. Machine validation came second. TypeScript, oddly maybe, appears to have a similar design goal.
I too used JSON Schema until it grated on me enough to finally abandon it... But I haven't taken the leap into TypeScript yet. Lately I've been specifying schemas with Tcomb[0], which I quite like. It's different from JSON Schema in that it's code and does runtime checking, so it's not a fit for every project. But it's pretty trivial to write a little node-based Tcomb validator if you want to be able to validate from a CLI. It's quite simple, but expressive, and I find that the fact that it's code can be a benefit, since it's simpler to compose complex schemas with code. As a bonus (or downside, depending on your use case), you also get pseudo-immutability on your Tcomb structs for free.
I do find it nicer to write my JSON schemas first as Typescript interfaces. Looks like there's a decent tool to automate that conversion to a JSON schema file [1], but I haven't tried that yet.
I also think it would be great if Typescript offered a "JSON strict interface" that could warn/error if you do something like use a Date instead of a string. (Even better if you could use that type information along with something like decorators for run-time checks before trying to `JSON.stringify`, at least in debug runs.)
I felt this pain when working with JSON APIs. I ended up writing my own literate JSON spec tool that looked like JSON but was in fact Python:
{"name": str, "age": int, "groups": [str]}
The template has the same 'shape' as the expected JSON, making it very easy for humans to grok.
We put all these specs into a file that acted both as documentation and as the actual Python code we used as templates to validate the real JSON. The doc was always accurate ;)
It turned into a full-blown dynamic type checking thing for Python, but it's still most useful for checking JSON APIs at runtime: https://pypi.python.org/pypi/obiwan/1.0.8
If you validate the JSON with an obiwan template, you can avoid having to type-check it as you parse it.
Obiwan syntax would be easy to use in other languages, but we never needed to try.
The structure represents the actual object, and comments allow for additional context/details... it's worked pretty well for documenting APIs that are under development, or intention when working with other devs.
Yes and with obiwan you can pass that structure to
data = json.loads(tainted, template=your_spec)
And that will throw an exception if the JSON didn't match the template. From then on its just to safely use the JSON you just loaded, knowing it is well formed and matches the expected shape... :)
At that point, you're inches from specifying protobufs or thrift, and the advantages in structured object parsing and tooling that exists from using one of those ecosystems is not to be ignored.
It's worth noting that depending on the language, JSON parsing is faster than binary implementations. Also worth noting that you can do JSON + gzip if there are larger records to be transferred regularly.
My attempt at this problem: http://www.teleport-json.org Currently working on a Haxe port so I can deliver this library to multiple languages from one code base.
JSON is a JavaScript child with all its pains and mistakes, and with one of the biggest ones of not looking into what's going on other platforms.
There was great work done by OASIS to look into all aspects of meta-modelling and data warehousing, that resulted in specs like MOF and UML2. I'd say that every modern DOM or other object metamodel must be based on these specs (or have some sensible mapping like the one done by Eclipse EMF project for XML schema to Ecore). What's good in MOF is that it easy to map it to programming language metamodels (for example, you can map MOF model to Java Reflection API).
Another option, if you don't need to design it from scratch, is just to use XML schema: JSON is basically a representation of DOM. In Java you can even use the same API to serialize both to JSON and to XML, which means that they both can be described by the same model language. If someone doesn't like XML schema syntax, it's probably possible to load XML Schema XSD into DOM and then serialize it to JSON - voila, you have JSON schema.
We wish a better standard would emerge for JSON Schema as well. We had to choose from just a few Validators that would grok the style of $ref we used - that part of JSON Schema is woefully underspecified. Of the few validators that we found, we found WILDLY differing performance and validation failure message quality.
The fastest JavaScript implementation we found was is-my-json-valid, but its error messages are nearly useless.
The best error messages (and IMHO conformance to the spec) came from fge's json-schema-validator, which is a Java implementation. The test suite on this impl is impressive. The validation failure messages are excruciatingly long and detailed if you have a large schema, but help a lot with debugging your document generators. But, the performance is awful. If you do any regex patterns, it shells to a Rhino script to to the validations which are predictably catastrophic for performance, making that validator unusable in production environments.
JSON Schema must be very difficult to optimize the way it is structured, which tells me it is probably a badly conceived specification.
JSON in general needs the kind of tooling and library love that XML got back in the day. I've searched in vain forever just for a great query language for JSON documents. I'm convinced that JavaScript+Lodash is the query language for JSON, to which I say meh. And, I'd love to see a first-class JSON streaming API and tooling/libraries (similar to STaX in the good old days of XML). How is anyone supposed to work with huge documents??
We're simply not giving ourselves enough tools, it's all very loosey goosey in JavaScript land.
Seriously. Everything the others can do, it does better (or, at least as good - not gonna argue that YAML tags are pretty); and most importantly, optionally. (Plus, it's got a few features the others don't)
You don't have to use block text. You don't have to use quotations. You don't have to use tags, or references. A simple usage of YAML is simpler than any other. A complex use of YAML is still less complex than many uses of XML, but has the same information content.
Try writing YAML in combination with templating (such as in Salt). Because indentation means hierarchy, it's terrible. I'd much rather use JSON in such a context.
I couldn't find an example of Salt on that site; more specifically, I couldn't find an example of the templating you're talking about. I didn't look very hard, but I'm spoiled by things like Sinatra, Coffeescript, and SlimLang.
Can you point me at an example of what you're talking about? I can't really evaluate what you're saying with the information I have.
This isn't to say that I would prefer JSON over XML - I most definitely wouldn't - but it is just as abused lately. I hope we will see its demise soon and something coming out of the YAML line (e.g. TOML) getting more widespread.
That's at the expense of a few things, however. For instance, array can't contain objects of different types; even [ 1, 2.0 ] is not allowed. Also, having a map (aka. a dictionary) in an array is really awkward to write, and there probably are cases that simply can't be expressed.
Thanks for the great article! It echoes most of my thoughts.
And thanks for Dotset - didn't know it exists, but it looks very, very neat.
In most but the most basic configurations, however, I found myself missing some of variables, inheritance, dates. TOML solves the last two, but as you point out it has limitations. YAML solves the first two, though in a bit of an ugly and obscure way. No wonder Salt and Ansible, two heavy users of YAML configs both pass it through templating engines to reach their goals.
For the lack of better alternatives, I usually end up with a Django-style Python module holding the configuration when I want it to be powerful. It still freaks (non-Python) people out though.
Lua is not a half-bad choice if you ask me. It takes some time getting used to, but it's fast to interpret, and at least one project that I know of - Lsyncd - is making an absolutely great layered configuration on top of it.
YAML & XML serialize / deserialize is slower than its JSON equivalent.
On top of that schema assisted JSON serialization can be almost as fast as something like protobuf, but you don't have to deal with binary or a lot of codegen on both sides.
Why? Is that a consequence of the specification, or an artifact of the implementations? Or, as in the case of XML, a consequence of common usage?
PS - I'm not counting the inherent different from simply more characters as important:
<obj>
<string>foo</string>
<obj>
obj:
string: foo
{ "obj": { "string":"foo"} }
That's not important. If you have enough extra characters for that to be important, you have enough data that you should probably do something other than serialize to text.
XML is significantly more complicated and creates a tree structure that doesn't map 1:1 to common programming strings, arrays, dictionaries and numbers. It creates nodes with attributes in a tree structure with namespaces and whole bunch more. The consumer parsing code is significantly more complicated too.
YAML is missing things such as quote marks so you have to do things such as type inference to determine what an item is, which decreases speed.
I don't know the specifics more than that, but just do serialization benchmarks and you'll see it pretty quickly.
The extensive optionality of YAML is probably also the very thing that has held it back and prevented it from ever quite breaking into the mainstream. It's very complicated, and yet, for all that additional complication, doesn't bring that much additional functionality. Options are not intrinsically good; they always incur a cost and the options must be worth more than their cost. To my eye YAML has always been designed with the premise that options carry intrinsic benefits rather than costs, which is a major design mistake.
You're talking about something like, but not exactly, choice paralysis?
It's possible that's part of what's going on, but... I don't think so. Or, not so cleanly. YAML options aren't choices, they're things you can do if you end up wanting to.
This is a theme in Ruby (and then Rails) programming. There's a bunch of stuff you can use, but you don't have to, and probably won't. There's an effect there, but it's not the same as choice paralysis.
--
I don't see what you see - that YAML was designed with the idea that options are intrinsicially valuable - it said, "Here's a thing to do. What's the simplest way to do this?".
Well-executed simplicity leads to options - think grep. There's a bunch of options when using grep, but really, it just does the one thing... but then there's a bunch of ways to use that one thing.
Again, this is a Ruby thing. There's more than one way to do it because there's more than one situation in which you'll need to. Have a simple array? Brackets. Have a not simple array? Dash.
In either case, it's pretty much the simplest way to do it, but one doesn't work (as well, or at all) for the other situation.
This is one of those duality things that comes up so often in programming and math: What is optional from the point of view of the user of the format is mandatory from the implementer's point of view. Pointing out that it's optional for the user does not remove the additional complexity the implementer must pay. YAML is a lot more complicated than JSON, and it's not all different ways of spelling the same thing (which is a sort of complexity that at least isolates itself to the parser), it's genuinely richer and more complex, and that price gets paid by somebody. Focusing on the one not paying that price doesn't make it go away.
Further, "YAML is not as popular as JSON" is an observational fact, not something we can argue away. It's worth trying to figure out why and learning what we can from it. I've made my proposal. There are other options ("Javascript is a steamroller"). But I'm not really a big fan of the evergreen "other people are just too stupid to see the obvious truth" explanation, which is generally the one that seems to be given to explain why YAML hasn't beaten JSON. That explanation is almost always wrong. (Only almost, though.)
You're saying that the popularity of JSON is directly related to the popularity of implementing JSON, which is due to the simplicity of it's implementation, as compared to YAML. This seems somewhat reasonable, but I don't think it can cover everything.
YAML supports tags, which could describe schema. JSON doesn't. Every method to describe schema for JSON will work for YAML, plus extras - like tags, and you could use node references to basically define types, and re-use them throughout.
Sometimes I feel like I'll go to Ruby hell for saying this, but as far as I can tell, XML is actually really good at that part... because it's a markup syntax, not a data syntax, while JSON and YAML are data syntaxes, not markups.
I use a lot of JSON and YAML via libvariant [0], a set of C++ libraries. e.g. the core class Variant is a JSON-ish object implementation; there are pluggable Serializers & Deserializers, etc.
Libvariant includes a command line tool, varsh ("Variant shell"), that can schema-validate JSON and YAML documents.
I am using https://github.com/keleshev/schema with python. I see a solid need for a schema here and there but don't buy into the whole heavy handed XSD. (Who hasn't struggled with the completely broken Microsoft .NET SOAP schema).
From a practical perspective I am tending to validate multiple sections of json with a schema closer to the code that uses it. This way I don't have to maintain some massive schema at the top.
(It is great for decorators
@validate_body(Schema({'size': Use(int), 'name': unicode})
def handler(body):
aa = body['size']
...
On a similar note, text protos actually work very well as configuration files and are used by SyntaxNet/Parsey McParseface [1] and Bazel's CROSSTOOL config [2].
By "just" I mean this is a simpler solution (compared to JSONSchema), not that it is an obvious solution. I'm pretty sure the author does not know about this solution since using Protocol Buffers for configs is not a well-known use case outside of Google.
107 comments
[ 3.0 ms ] story [ 183 ms ] threadhttp://jschema.org/
The idea was/is to capture the spirit of JSON minimalism, avoiding the insanity of XML-based schema languages.
Currently, some UCSC students are working on javascript and java tools to work with schemas. We should be able to release that work in the next month or so.
And there is (or, better say, was) an "Orderly JSON" language that had nice and compact human-manageable representations, but had essentially died from nobody using it.
JSchema:
Pros: is JSON, is simple to read and write. Cons: is (currently) limited to very primitive schemas.JSONSchema:
Pros: is JSON and reasonably powerful. Seems to be a most common standard out there, with lots of libraries available. Cons: is verbose to the extent it's no fun to read or write by hand. Has some ambiguities and - as the OP article correctly says - spec is unfinished.Orderly JSON:
Pros: compact and powerful. Cons: not JSON but a custom grammar, no one uses it.1 - http://blog.cognitect.com/blog/2014/7/22/transit
I can appreciate that an order-independent representation can make schema versioning easier, but what other advantages does this provide?
Python dictionaries hold keys but the keys aren't in any particular order, so allowing them to come out in whatever order maps very naturally to the language.
The worse offender, imho, is Microsoft's DataContractSerializer, whose default behavior is to silently ignore this ordering failure and also silently fail to deserialize any out-of-order elements.
Either fail loudly or be flexible about ordering.
There is no well-adopted analog to xmlschema for json.
https://github.com/fge/json-schema-validator/issues/182
i refuse to use XML just like everyone else, but not because it's hard to parse data from.
Although stringly-typed XPath has the same problems as SQL, building complex queries through string concatenation. This is where JSON shines, because you usually just convert to a tree of your language's native data structures in one line and use your languages native facilities to traverse it from there.
object.listOfItems .reduce((item) => item.type === 'car') .map((item) => item.name));
This is way more powerful than xpath.
The advantage of a schema is you can validate your data and move to static typing in only a few lines of code. It's for this reason that I think the best option these days is to just use something like protobufs v3, which has a stable JSON serialization and a sensible schema specification.
But even for objecty data, rather than "documents", XML doesn't have to be clumsy. Checkout the code samples here for C++
http://www.codesynthesis.com/products/xsd/
The really good thing about XML, that it - unlike JSON - is extensible.
And traversing both JSON and XML (without the weird parts) are no-brainers. Every (or about so) mainstream language has libraries that make either format really transparent to use.
That's really only true about XML, that is not the case with JSON. JSON is a pain to work with in C# for example compared to XML.
I did some toy project in C# and haven't found handling JSON any painful. Maybe I just haven't hit the bad cases, though.
I don't exactly remember what I've used and what I wrote, but I think I had just installed some library (IIRC it was Json.NET), put some annotations on classes, and got my serialization. For deserialization I didn't wanted to write throwaway classes so I just made a JSONPath query and iterated over the result objects, looking at necessary properties. Nothing particularly different from how things are in, say, Python, except maybe for some extra type checking.
[0] https://en.wikipedia.org/wiki/YAML [1] https://en.wikipedia.org/wiki/INI_file
If you do not care too much about exchanging data with other apps then parser/generator supporting javascript style comments (e.g. json-cpp) would work. With each JSON value it can store "commentBefore" and "commentAfter". If you need you can strip these comments any time by parse + write cycle with "collectComments" option inactive.
And I find the whole "run minifiy/strip first" to be a hilarious suggestion from Douglas Crockford. It can be rephrased as "If you want comments in JSON, for a config file say, then don't use JSON."
That, and XML requiring the tag name in the closing tag, are some of the obvious, silly, mistakes. XML in particular. Without the named closing tags, just using, say, <>, would reduce the apparent bloat and annoyance by a very large factor.
See: https://github.com/edn-format/edn
Unlike XML and UML, there's just less of a market for GUI-like editors for JSON?
Something that would also be nice, if anyone comes up with such a beast would be the ability to edit a multi-record json file... effectively each line is a single JSON, and an LF at the end of each line/record. I've used this structure a few times. Given that JSON will typically encode special characters allowing for single-line representation by default, it's worked really well, especially combined with gzip as a stream for archiving data.
If anyone knows what that program was, would love to find it again.
Tweeted it... It's funny, but I often use twitter to tweet about stuff, just so that I can find them later.
Opening a one-line JSON file of a few MB in Emacs can be painful. The editor struggles to handle the long line and the matching-parenthesis-highlighter runs its stack machine down the whole length looking for the closing delimiter.
There are ways to avoid this, like pretty-printing the JSON across multiple lines, or forcing fundamental-mode when large files are opened, but I still haven't set these up as default.
I believe the question was about good editors for json documents.
On the other hand, I did a lot with XML in the 00s, and I sure don't want to go back to dealing with that anymore. I'm sure the world has come a long way since SAX was the thing to use for XML, but that experience left me so deeply scarred that I'm willing to just write an ad hoc schema on my frontend and say, "after JSON.parse, go in and coerce these fields to be int / bool / Date, etc". Not great for interop, but what the hey.
Are you thinking something like
Or were you thinkingThe other option ("#date#") is bad, IMO, since it's a magic string. I don't like magic strings because they blow up at unexpected times. Imagine the notice to the users: "please stop using the hashtag #racecar#, we know palindromes are cool, but it is causing our database to crash". I had similar issues with CDATA blocks in XML, actually - escaping strings to afford reasonable parsing is never fun.
On the other hand, the date literal you suggest is not part of javascript. And on that note, why not just use timestamps for dates in json?
With JSON.parse()? That would be a big failure of handling standard JSON if it did that. Do you have an example where JSON.parse() doesn't correctly parse JSON in the manner you suggest? Or really in any manner? I haven't seen it fail yet and if there are known failures it would be good to know about them.
https://jsfiddle.net/jb1wunpc/
I don't know where the issue really originated which part of transport or serialization, and I honestly don't know how common it really is - I just slapped some type coercions on the object I was putting into the jsonb field (which was necessary anyway, to protect against malicious user input) and called it a day.
So it's probably not JSON.parse at fault here.
EDIT to add: just looked over my notes, and it seems like the problem was likely originating in how I was treating option elements underneath a select in an html form. Somehow the value was getting set to a "10" instead of 10 on the form's model object, and that was getting pushed over the wire (serialized and deserialized) as a string.
But then I realized that what I had made up was very similar to type definitions in TypeScript!
Especially if you use `interface` and not `class`, TypeScript gives you a lot of flexilibity in specifying data structures. You get optional values, typed arrays, typed but mixed arrays, references to other named types that you define yourself, and so on. Really, it's all you need and more. The only downside is that you can spec stuff that you can't put in JSON, such as Date objects. As a schema language, that is a bit weird. But just don't spec it like that! :-D
I'll post some examples in a reply when I'm back behind a computer.
[0]https://github.com/eteeselink/relax-json
For the record, JSON Schemas are quite a bit more expressive than TypeScript interfaces, or even Scala Traits. I actually put together a big list of every JSON Schema constraint that isn't checkable at compile time in TS: https://github.com/bcherny/json-schema-to-typescript#not-exp....
Fair enough on the constraints. One of the design goals I had at the time was that I didn't want stuff like that to be expressible. Basically, it felt over the top to me - you might as well just allow writing predicates in plain JS then. Even more expressive!
My primary goal was a spec language for humans to read and write. Machine validation came second. TypeScript, oddly maybe, appears to have a similar design goal.
[0] https://github.com/gcanti/tcomb
I also think it would be great if Typescript offered a "JSON strict interface" that could warn/error if you do something like use a Date instead of a string. (Even better if you could use that type information along with something like decorators for run-time checks before trying to `JSON.stringify`, at least in debug runs.)
[1] https://github.com/lbovet/typson
We put all these specs into a file that acted both as documentation and as the actual Python code we used as templates to validate the real JSON. The doc was always accurate ;)
It turned into a full-blown dynamic type checking thing for Python, but it's still most useful for checking JSON APIs at runtime: https://pypi.python.org/pypi/obiwan/1.0.8
If you validate the JSON with an obiwan template, you can avoid having to type-check it as you parse it.
Obiwan syntax would be easy to use in other languages, but we never needed to try.
There was great work done by OASIS to look into all aspects of meta-modelling and data warehousing, that resulted in specs like MOF and UML2. I'd say that every modern DOM or other object metamodel must be based on these specs (or have some sensible mapping like the one done by Eclipse EMF project for XML schema to Ecore). What's good in MOF is that it easy to map it to programming language metamodels (for example, you can map MOF model to Java Reflection API).
Another option, if you don't need to design it from scratch, is just to use XML schema: JSON is basically a representation of DOM. In Java you can even use the same API to serialize both to JSON and to XML, which means that they both can be described by the same model language. If someone doesn't like XML schema syntax, it's probably possible to load XML Schema XSD into DOM and then serialize it to JSON - voila, you have JSON schema.
The fastest JavaScript implementation we found was is-my-json-valid, but its error messages are nearly useless.
The best error messages (and IMHO conformance to the spec) came from fge's json-schema-validator, which is a Java implementation. The test suite on this impl is impressive. The validation failure messages are excruciatingly long and detailed if you have a large schema, but help a lot with debugging your document generators. But, the performance is awful. If you do any regex patterns, it shells to a Rhino script to to the validations which are predictably catastrophic for performance, making that validator unusable in production environments.
JSON Schema must be very difficult to optimize the way it is structured, which tells me it is probably a badly conceived specification.
JSON in general needs the kind of tooling and library love that XML got back in the day. I've searched in vain forever just for a great query language for JSON documents. I'm convinced that JavaScript+Lodash is the query language for JSON, to which I say meh. And, I'd love to see a first-class JSON streaming API and tooling/libraries (similar to STaX in the good old days of XML). How is anyone supposed to work with huge documents??
We're simply not giving ourselves enough tools, it's all very loosey goosey in JavaScript land.
Seriously. Everything the others can do, it does better (or, at least as good - not gonna argue that YAML tags are pretty); and most importantly, optionally. (Plus, it's got a few features the others don't)
You don't have to use block text. You don't have to use quotations. You don't have to use tags, or references. A simple usage of YAML is simpler than any other. A complex use of YAML is still less complex than many uses of XML, but has the same information content.
Can you point me at an example of what you're talking about? I can't really evaluate what you're saying with the information I have.
This isn't to say that I would prefer JSON over XML - I most definitely wouldn't - but it is just as abused lately. I hope we will see its demise soon and something coming out of the YAML line (e.g. TOML) getting more widespread.
I only skimmed the Github readme, what makes TOML better than YAML?
That's at the expense of a few things, however. For instance, array can't contain objects of different types; even [ 1, 2.0 ] is not allowed. Also, having a map (aka. a dictionary) in an array is really awkward to write, and there probably are cases that simply can't be expressed.
It has first-class dates, though.
I wrote about the state of settings formats around the time that TOML was born: <http://espadrine.tumblr.com/post/44576550326/a-history-of-se....
And thanks for Dotset - didn't know it exists, but it looks very, very neat.
In most but the most basic configurations, however, I found myself missing some of variables, inheritance, dates. TOML solves the last two, but as you point out it has limitations. YAML solves the first two, though in a bit of an ugly and obscure way. No wonder Salt and Ansible, two heavy users of YAML configs both pass it through templating engines to reach their goals.
For the lack of better alternatives, I usually end up with a Django-style Python module holding the configuration when I want it to be powerful. It still freaks (non-Python) people out though.
Lua is not a half-bad choice if you ask me. It takes some time getting used to, but it's fast to interpret, and at least one project that I know of - Lsyncd - is making an absolutely great layered configuration on top of it.
On top of that schema assisted JSON serialization can be almost as fast as something like protobuf, but you don't have to deal with binary or a lot of codegen on both sides.
PS - I'm not counting the inherent different from simply more characters as important:
<obj> <string>foo</string> <obj>
obj: string: foo
{ "obj": { "string":"foo"} }
That's not important. If you have enough extra characters for that to be important, you have enough data that you should probably do something other than serialize to text.
YAML is missing things such as quote marks so you have to do things such as type inference to determine what an item is, which decreases speed.
I don't know the specifics more than that, but just do serialization benchmarks and you'll see it pretty quickly.
It's possible that's part of what's going on, but... I don't think so. Or, not so cleanly. YAML options aren't choices, they're things you can do if you end up wanting to.
This is a theme in Ruby (and then Rails) programming. There's a bunch of stuff you can use, but you don't have to, and probably won't. There's an effect there, but it's not the same as choice paralysis.
--
I don't see what you see - that YAML was designed with the idea that options are intrinsicially valuable - it said, "Here's a thing to do. What's the simplest way to do this?".
Well-executed simplicity leads to options - think grep. There's a bunch of options when using grep, but really, it just does the one thing... but then there's a bunch of ways to use that one thing.
Again, this is a Ruby thing. There's more than one way to do it because there's more than one situation in which you'll need to. Have a simple array? Brackets. Have a not simple array? Dash.
In either case, it's pretty much the simplest way to do it, but one doesn't work (as well, or at all) for the other situation.
Further, "YAML is not as popular as JSON" is an observational fact, not something we can argue away. It's worth trying to figure out why and learning what we can from it. I've made my proposal. There are other options ("Javascript is a steamroller"). But I'm not really a big fan of the evergreen "other people are just too stupid to see the obvious truth" explanation, which is generally the one that seems to be given to explain why YAML hasn't beaten JSON. That explanation is almost always wrong. (Only almost, though.)
You're saying that the popularity of JSON is directly related to the popularity of implementing JSON, which is due to the simplicity of it's implementation, as compared to YAML. This seems somewhat reasonable, but I don't think it can cover everything.
Sometimes I feel like I'll go to Ruby hell for saying this, but as far as I can tell, XML is actually really good at that part... because it's a markup syntax, not a data syntax, while JSON and YAML are data syntaxes, not markups.
Libvariant includes a command line tool, varsh ("Variant shell"), that can schema-validate JSON and YAML documents.
[0] https://bitbucket.org/gallen/libvariant
There's https://tools.ietf.org/html/rfc7159, co-constraints https://www.ietf.org/internet-drafts/draft-cordell-jcr-co-co..., compact syntax
On a similar note, text protos actually work very well as configuration files and are used by SyntaxNet/Parsey McParseface [1] and Bazel's CROSSTOOL config [2].
[1] https://github.com/tensorflow/models/blob/master/syntaxnet/s...
[2] https://github.com/bazelbuild/bazel/blob/master/tools/cpp/CR...