Specifically, lines that begin with a word or words followed by a colon are dictionary items; a dash introduces list items, and a leading greater-than symbol signifies a line in a multiline string. Dictionaries and lists are used for nesting, the leaf values are always strings.
No doubts that there are use cases for this, but calling something that casts everything to strings an alternative to the above formats seems like a bit of a stretch.
It's not a bad idea, as text file formats are all strings anyway, to let the application do the conversion. It's the one with the domain knowledge anyway. Perhaps it could do with a companion library to specify the data format and emit parse errors as appropriate. But to keep that out of the syntax makes a lot more sense than the insanity that is yaml.
I've been using toml for my latest project and the one thing that really bitten me is that [1,2,3] is a valid array [1.1, 1.2, 1.3] is a valid array, but [1, 1.5, 2] isn't valid and throws an exception due to it having heterogeneous types.
This is something that was fixed in the forthcoming TOML 1.0 spec. Parsers that have been updated to support it will allow heterogeneous types in arrays, which your application can convert to a vector of floats or whatever collection type it uses internally.
I’m not sure the world needs another sexpr/sgml isomorphism, though it does look pleasing to the eye for the given examples. What this doesn’t solve though: yaml+jinja use cases, code as text, schemas outside of the document, everything else that makes a language out of a syntax tree.
> For example, in JSON 32 is an integer, 32.0 is the real version of 32, and “32” is the string version. These distinctions are not meaningful and can be confusing to non-programmers.
I'm really struggling with this assertion; IMHO one of the problems with JSON is the lack of more sophisticated scalar types.
That being said, this appears extremely readable, so my concerns could definitely be alleviated by a decent schema.
In the design of YAML, Ingy made the case that we shouldn't have types for scalar values, that they should all just be strings. His argument was that each application knows what it needs and should have the ability to direct how those scalars are processed. If the format needs to be standardized across applications, one can use a schema system layered on top. In retrospect, I think he was right and I was wrong. For example, Fast Healthcare Interoperability Resources (FHIR) serialized as JSON treat numeric values as decimals, e.g. `2.3` is not a floating point value. Moreover, since parsing numbers is slow, it should be deferred till you actually need to interpret the numeric value.
It's less of a problem if you're using it for configuration files, where a program knows what key's values need to be cast to an integer or float.
But it seems disastrous if you wanted to use it for storing or transmitting data, above all between applications. You're immediately throwing out the possibility of being able to serialize and then deserialize data in basically any programming language.
I shudder at the idea of an API that accepted NestedText, where I'd need to worry about whether my floating-point output was compatible with its floating-point string parser. Yikes. I want the serialization format to handle that. Isn't a major criticism of JSON that it doens't have a built-in datetime representation?
You wouldn't use this for arbitrary object serialization. Lists can only contain strings, not other lists or dictionaries.
As for stringification, JSON's data types are mismatched with pretty much everything that isn't JavaScript to some degree. If you need to serialize a 64-bit integer to JSON, you serialize it as a string because the parser on the other end is probably going to try to parse it as a double-precision floating point number. Once you've started serializing numbers as strings anyway, it's not too far to "serialize every scalar as a string".
Ah ha, got it. The page never really makes it clear that this is for configuration files only, not serialization.
Indeed, not being able to have lists of dictionaries, or lists of lists, is very restrictive. Seems to be for very simple configurations only. E.g. a set of preferences, but not a set of monitor calibrations. (Inventing arbitrary dictionary keys seems pretty hacky.)
Well, there's nothing about the JSON spec that says you can't serialize 64-bit integers to JSON. The grammar allows unlimited precision. Implementations may vary.
> Lists can only contain strings, not other lists or dictionaries.
I don’t think this is true. None of the examples have lists containing non-string objects, but the documentation doesn’t seem to draw a distinction between lists and dictionaries wrt what can be placed in them.
(Both lists and dictionaries are initially described as only containing strings, and later this description is expanded to include nesting; this counterintuitive arrangement may explain the confusion.)
To be fair, this seems to be designed specifically for config files. Just as JSON should be used for data transmission but not for config, I could imagine, this should be used for config but not for data transmission.
What if someone inputs 2.2.5 when the code is expecting an int? Or “abc”? It seems like it’s pushing all the config validation into user code which sucks.
The code already has that problem. Certainly, if you're using YAML, a user could type 'version: 2.2.5' where you're expecting a major version number (an int), and all of a sudden your code is passed a string instead (or a string-flavored variant). You can imagine the same sort of problem in JSON too, usually from someone leaving off a quote where you're expecting a string. NestedText's philosophy seems to be, you're going to need your code to handle this anyway, we'll just pass you a string in all cases and it's up to you to convert it to an int on your own and validate it.
Frankly, in most languages, this is better because you don't have the types of objects randomly change based on user input. (In a few languages, with a few libraries, you can specify the type of the document to the parser and have it fail to parse the entire document if it can't deserialize to the right type, in which case this is a little weaker. But you can still do that with NestedText, just one step after the parser - have your own function that takes a ComplexStructure<..., String, ...> and returns a ComplexStructure<..., int, ...> or throws an error.)
Only in untyped or dynamically-typed languages. But even in JavaScript one may write +obj.version instead of obj.version to make it numeric. Evading this is a straight way to hell.
In a statically typed one, conversion is typically generated from description and type checking applies just at reading.
The problem with vaguely specified format is in more simplex cases. Shall we accept 45x as number (and what it value will be, 45 or 0)? 045? 045x? What date is 1/2/3, 1-2-3? And so on.
> Shall we accept 45x as number (and what it value will be, 45 or 0)? 045? 045x? What date is 1/2/3, 1-2-3? And so on.
I think the point is that the answers to those questions may well be application-specific. In which case it is better to not bake them into the file format.
In a statically-typed language, this is an entirely reasonable thing to want, but also, existing languages like JSON and TOML don't give you that either:
- There's no way to say that you want an integer; you get a floating-point value.
- Someone can always specify something of the actual wrong type. (Imagine changing YAML "version: 1.9.1" to "version: 1.10". You can't just stringify 1.10, you'll get "1.1"!)
So, in a practical data format, the schema for your document needs to say something like "This is a number, which must be an integer between 0 and 2^16" or "This is a string, make sure to quote it" or whatever, and a generic statically-typed JSON- or YAML-parsing library isn't going to handle that for you. And telling your users "the input format is JSON" doesn't answer that question: you must make it explicit to users.
Fortunately, you can handle it just fine in a statically-typed language in one of two ways. One is to accept an object from your parser that consists of variant types and pass it through your own function that validates it against a schema, and then either returns a more-restrictively-typed object or throws an error. Such a function could easily do string conversion too if given NestedText input, as I mentioned. The other is to pass some information into your parser saying, don't act like a generic JSON/YAML parser, instead interpret these particular fields in this particular way and accept only things with this structure. If you're doing that, you can easily tell the parser to use this particular string-to-integer function on the strings in NestedText and then return an appropriately-typed object containing an integer to you.
I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation converts numbers to doubles, another to either doubles or longs depending on whether they have a . or e, and another which converts them to bignums (or passes them as strings to the caller).
One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
There are also issues when something looks numeric but doesn’t parse (eg 1.2.3, 3/2, 12in, 4h30m2s, 2:30, 2020-02-29, etc). One way to deal with these is a tokenisation rule like in Common Lisp: if it is a valid number syntax then treat it as a number, otherwise it’s a symbol, but this can lead to issues (eg you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes. It seems crazy to pass that detail on to the poor sod who has to write the config file).
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.
Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false".
The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.
Except YAML is straight-up wrong at times, unless you know all the edge cases. I just learned unquoted NO is coerced to False. A classic case of leaky abstraction/"bad magic".
Unless you have actual type annotations/tags (eg xml, jsonld, graphql), everything IS a string. There's no assumptions otherwise.
There isn’t really any magic in Perl, just lots of unfamiliar lexicon if you come from more traditional programming languages. Perls problem is it’s history is rooted in command line usage so there’s a tonne of inherited reserved single character variable names and such like that are optimised for keystrokes rather than readability. However you can certainly write Perl programs that don’t follow those older conventions and look more like a modern language.
That kind of magic in Perl has a different kind of meaning than "magic number/bool/string parsing" in YAML.
Perl tied-variable magic just means there are (effectively) getter and setter properties attached to the variable. "Magic" is just the name that was chosen in the implementation, and it stuck.
It's used to implement variables with special, automatic meanings, like $$ for "current pid" and $! for "last error".
It's also used to implement variables with user-defined behaviours on access, which is quite handy for a lot of abstractions.
A lot of modern languages support both of these things, because they are useful, but it's not called magic in those languages, it's called something like "watchers", "proxies", "getters and setters" or "hooks".
No, the criticism of YAML-style "magic" is that it leads to entirely surprising behaviour from innocuous input. Perl magic is not that kind. If you're using a special variable, you already know why.
That’s just playing around with the same reserved variables I spoke of before and there is nothing magical about them aside their silly name. In fact the opposite is true, they’re actually predictable and well documented. They just happen to have terse names as a throwback to command line usage (eg you wouldn’t call $1 a magic variable in Bash because it happens to work the same as ARGV[1]).
In fact in Perl, you can opt for longer, readable, lexicon over the terse single character variables; and that’s literally how modern Perl should be written.
Whereas the problems described with YAML is where it can automatically alter your data based on what the parser “thinks” the data should represent. Which is generally what people mean when they talk about “magic” in IT: systems that don’t honour your input and instead automatically convert it into something else. Perl doesn’t do this even in spite of it looking like executable line noise to many.
I think "magic" is often just abstraction. And while abstraction is certainly necessary to speed up work and declutter the brains of the end users, too much abstraction takes control away and bad abstraction takes the wrong decisions in your name.
If you take a look at how string handling works in most programming languages there is a lot of "magic" going on there. Which isn't a bad thing necessarily, because most programmers don't want to deal with the intricates of strings unless they really have to. The key is that this magic doesn't get in your way and doesn't do too magical things nobody ever asked of it.
That seems like a pretty sensible approach - eliminates the "Norway problem" altogether. I'd guess you have to use the "%YAML 1.2" directive in all documents in order to get it though...
You don’t need this directive to get YAML 1.2 parsers to parse your document as YAML 1.2; that’s the default. You only need it to instruct YAML 1.1 parsers to raise a warning (and I guess for parsers of future version of YAML to fall back to 1.2 behavior, if that ever happens).
Can't say I love the notation, but indeed that is type annotations. I guess neither "yaml type hints" nor "yaml type annotations" are the right query. Had to search explicitly for "yaml tags".
The neat thing is that you can use tags to represent custom data structures or functions. For example, AWS Cloudformation uses YAML tags as a shorthand for template functions [0].
I think this is an “input format” vs “serialization format” issue. NestedText is meant for user input so it’s focusing more on the UX of the syntax. In that case, having to parse strings is a reasonable task for developers to take on
(Consider what happens if you try to send a tweet's ID, a perfectly normal number like 205052027259195393, through JSON. Or if you try to serialize a stack trace on a 64-bit system, where addresses are also perfectly normal numbers.)
The example uses numbers outside the range of JS numbers (too many significant digits), so when the JSON is evaluated as a JS number some of the least significant bits are rounded. Personally, I think it is a mistake to use values in JSON that can’t be represented in JS, but the standard doesn’t explicitly forbid it.
It's more a JS thing: The lone JS number type is double-precision floating point, meaning beyond a certain range there are integers that cannot be accurately represented as a JS number.
The JSON standard doesn't place restrictions on size or precision of numbers, instead just noting that implementations can vary their treatment of and limits on numbers. While JS uses doubles for all numbers, many other languages emit an integer type for a JSON integer. So, once you go beyond the range where a double can accurately represent all integers, you run the risk of a mismatch in how the number is interpreted by different languages parsing the same JSON.
Of course the spec also allows you to create way too big or too precise numbers that would be problematic in most languages as well; it's just that this is a somewhat common bugbear.
I wouldn't necessarily call it a flaw in JSON though, more an issue with JSON.parse or really just a fact of life when dealing with numbers in JS. Alternatives to the built in JSON.parse exist to read large integers as strings or bigints.
It's a Javascript problem. Integer caps out at 9007199254740991 (253-1), and after that it's treated as a BigInt. As long as you're working with BigInt literals, it'll be accurate, but when you convert back and forth you can lose precision.
JavaScript is using double as a number. There's no such thing like integer or floating point numbers per se, only so called Number.
Described issue is not problem of a JSON but engine which parsed it and language which stands behind the parser. Any config format will eventually have same result and same issue.
So forcing programmer to parse every single piece of data for sake of "it's his responsibility" is not a case here.
I also disagree this is in any way programmer responsibility to create standardized way of creating parser for everything. This format gives you nothing but indentation so you are forced to create documentation for everything field, what type it's and what kind of values it takes. Lots of extra work for nothing when you have any other format.
Why wouldn't you? It's a number. It's also sorted (up to about one second of inconsistency between Twitter worker processes on different machines), and you want to sort them by numeric sort, not by lexicographic sort as you would with strings.
I mean, there's a pretty clear argument here: Twitter themselves used to return these numbers as numbers in the API until they realized they were about to hit this problem. https://developer.twitter.com/en/docs/twitter-ids
Using numeric types as ids probably leaks into other areas, where being numeric is then assumed. When you want to change it at some point, you got to be very careful suddenly. A string could simply contain a new kind of id. Seems less refactoring effort would be required. On the other hand, switching from numeric to strings might give you compiler errors when types do not match, so perhaps it might even make refactoring simpler.
You can always add a second or third key. Stop messing with the primary key. Many systems have a numeric primary key and then only expose a hash or UUID to the public.
> The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.
I don't think that's what this tool is for. This tool is for humans to read and edit data. That's a different use case from automated programs exchanging data, for which I agree you should be using standardized numeric and boolean data types and not making everything a string. But how that standardized data gets determined from data that humans enter should be up to the individual application.
What is the use of this then? Humans don’t need structured data to exchange information and if this is insufficient to communicate it to the machine then it won’t be a great human to machine format either.
> Humans don’t need structured data to exchange information
Maybe we don't need it, but it often helps.
Also, this data format isn't necessarily for humans to exchange data with other humans, but for humans to give data to applications in a format that's much easier for humans to use.
> if this is insufficient to communicate it to the machine
Not at all. Each specific application can easily parse this data format according to its own needs. What this data format doesn't specify is a single translation into application data that is the same for all applications. But applications don't need or want that, because they have different use cases.
I see. That makes sense. Now that I think about it maybe if its in front of some database or storage that only handles strings it would work well as well.
It is still useful for humans to have a well-understood way to input typed data.
If you make everything a string then the interpretation of "no" as boolean true or false is left to each tool, and there are even tools which have different interpretations of "yes/no" for each field.
If NestedText is meant to be written by non-programmer humans, the details of null, undefined, 0, false, "NO", nil, or whatever else, would likely also be lost on them.
Most likely if their input is meant to be machine interpreted they would need to be trained to provide specific inputs anyway. I like that NestedText doesn't hide that problem. It lets the user organization decide how it wants to manage that problem, and what symbols or words are understood by the people authoring the files.
That's not the only way to deal with them, you can specify type like bson.
The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined.
The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.
My ideal config language would have unambigous syntax for at least 64-bit signed integers and doubles (so, for example the spec for the language requires 56 to be parsed as an integer and 56.0 to be a double. Additional types would be ok too, as long as the syntax is unambiguous and obvious.
The spec requires that a full 64 bits of signed integer be parsed and understood, that hexadecimal, octal, and binary values be integers, and that floating point values be parsed as doubles.
It doesn't support hexadecimal float, however, which is a pity: having a guaranteed bit-identical format is a nice affordance.
I am of the opposite view: As long as there are no edge cases, supporting more types in data languages is good, it leads to failing faster.
Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot.
> you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes
Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that)
(disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda)
> I think avoiding numeric types is a good decision.
Only if this format is intended for use-cases that never need to deal with numbers.
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
That statement couldn't possibly be more wrong.
Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following:
- Unsigned 64-bit numbers.
- A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error?
- NaN
- Infinity
- Negative zero
- Denormal numbers.
- Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats!
- Efficiently encoding floating point to use the minimum digits without losing precision.
- Parsing those minimal numbers with perfect "round-tripping".
- Doing the above efficiently.
- Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser...
The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient.
Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that.
Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit!
But which of these are problems in configuration files written by a human? That is the aim of the format. Moreover in applications were there could be issues, it would most certainly be tied to very specific fields and you would want specific application logic to handle that field. Now if people misuse it as a data exchange format or so, yes I agree with you, but at that point just use a binary format instead.
That doesn't matter at all. The author's aims will be ignored if this format is used for anything even vaguely important. Eventually it'll need tooling to both read and write it.
DevOps pipelines, applications with GUIs, or something will need to both parse and generate this format in a consistent way.
There is no such thing as a human-write-only format in widespread use.
Even programming languages are regularly generated by tools such as RPC API codegen tools, LINQ-to-SQL and the like.
You’ve given lots of examples of things that make parsing numbers difficult but I don’t see why they are relevant to a config file written by humans. I think it makes sense to have the number parsing owned by the thing which cares about the number format.
One example you provide is decimals for currency values but I claim you would want such values to look like $1234 in config files so that when they are reviewed or written, the person reading the file knows they are looking at a dollar value and can be concerned if it is too large.
I’m not suggesting that applications write their own number parsing. Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.
> Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.
Okay, so the computer is doing the parsing.
Those functions are notoriously inconsistent in their behaviour, particularly across different programming languages. If you're not careful, you'll end up accidentally using the internationalised versions of those functions. Even if you're careful, other people won't be.
Remember, data formats are for interchange. They have to be language agnostic. They have to be well-defined, and it should be possible to write a parser for them without having to guess at the precise details.
I wrote some config for my application, which knows how to read it. Why do I want some other application In some other programming language to read it too?
I am far more worried about localisation issues than language issues. If you are storing something central to multiple applications I'd argue a text file is the wrong tool
If you go fully against Robustness principle, you lose the reason to use textual formats as well, since they are designed to be forgiving of human errors in input and catch them in syntax.
And - it is certainly OK in many instances to have fixed-width, fixed byte-order binary encoding as the format's basis. It comes with the twin downsides of wholly different categories of errors cropping up, and with the lack of a universally agreed upon tool for human entry.
Perhaps text was a fashion, though. I definitely have had thoughts in that vein lately. And in that case we shouldn't always be rushing to use it as the source of truth when we have many good, machine-level agreements about numeric formats.
> I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation (...)
It seems you're mixing up the language definition with implementations that try to follow the language definition.
If different implementations have different results then either they are buggy or the language has some important holes in the specification.
Either way,the solution to this problem is not less validation.
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
That's the whole point, isn't it?
I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions?
Wouldn't it simply easier to let the language and the parser do that already?
I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.
It works fine in Python. It's nice because the visual indentation lines up with how the interpreter actually reads the code. You don't end up in a situation like this contrived example:
if (something == somethingElse)
doSomething();
--counter; // added afterwards
when whitespace has actual meaning.
Your editor adds the whitespace to match the current scope, so you don't have to type it manually.
I don't mind it that much for programming languages like Python. The one annoyance I have is related to editor tooling: In languages that use braces to denote blocks, I can paste a block of code wherever and have my editor autoformat it to the correct indentation level, while in Python I have to be more deliberate with increasing or decreasing the indentation of a pasted snippet, and sometimes I'll make a mistake in the process that isn't caught until later.
I dislike significant indentation for configuration because the nesting tends to get deeper than for programming languages and it can be harder to see what's going on.
> You don't end up in a situation like this contrived example:
Modern languages repair this with mandatory block braces: {C style} (Rust, Go, Swift...) or Modula/Ada style (if x then foo else bar end). C style is easier for editors.
> What's your issue with significant whitespace?
1. Many cases when leading whitespace is still spoiled, including editors (which use can't be avoided due to corporate tool limiting), web formatters, etc. Last decade spreading of mobile-aware tools made this worse.
2. With grouping by indentation, it's hard to determine block start if multiple blocks are ended at the same line, like:
a:
b
c
d:
e
f
p
and let's imagine the most nested block occupies 2-3 screens (not rare for configs or code, even if it is against good coding rules). You have no means to decide what block start is to be matched when you ask editor to find match.
I code in Python continuously since 2004 and I deem grouping by indentation is bad idea. Not fatally bad, of course, but with explicit grouping it would be a bit better.
Why not? I'm not a Python developer, but bad indentation is considered a faux pas in most production codebases and is already enforced by linters, so I don't really see the issue with enforcing it at the language level.
> With NestedText any decisions about how to interpret the leaf values are passed to the end application, which is the only place where they can be made knowledgeably. The assumption is that the end application knows that Enrolled should be a Boolean and knows how to convert ‘NO’ to False.
That assumption is... not applicable to most scenarios I come across and will likely lead to issues being pushed downstream and introduction of subtle bugs and defects.
It'll lead to incompatible NestedText codecs/serdes :(
There should be at least some support for some standardized representations (basically JSON + ISO8601 datetime + some encoding for embedding whatever stringified serialization, eg. just how HTTP uses chunks and unique boundary tokens).
Interesting insights into the thought process of the author, there's very little as to positive statements of why you'd want to use this format, but they do however spend a great deal of time discussing neutral features and edge cases of currently wildly adopted alternatives.
One might call this 'argument by gotcha' interesting to imagine the kind of lifeworld that makes one think this is a worthwhile form of persuasion.
I like it. For use cases that are for non technical users to manipulate and maintain it makes sense to me.
In Emacs, org files are often used for configuration similarly.
While I'm a fan of EDN most of the time, which doesn't suffer from the issues they mention JSON having, not having to wrap things in quotes for text, and using indent for nesting definitely would make it nicer for non technical people. So I can see a use case for this as a simple text interface for forms that is targetted at non-technical users.
You know what I want? Schemas. And clear error messages.
I want to know beforehand what I can put in a config file and I want a fast and hard failure if what I put in there is not good.
And this should be implemented at the file format parser level, with hooks for apps to add on top of the default behavior, so that every app that implements this format gets these things almost for free.
Like in Windows where you configure by clicking check boxes that can get disabled if invalid, with tooltips explaining what they do, additional help if you press F1, etc. ?
Haven’t you described cap’n’proto, protobuf, thrift, flatbuffers etc?
I know cap’n’proto also has fantastic support for using the schema for config files. You can just compile any constant as a stand-alone serialized message that you mmap into your code in a safe way. It can’t do complex math and things (at least yet) but you can express lists, dictionaries, and reference other constants, so as a config file replacement I love it. I’ve also found the format to be far more regular and consistent than you get with things like text protobuf (you’re still using the schema language instead of another format)
I think the parent is typing to say that the data is stored in a map which is read to a proto, etc. Kinda like what GPRC does over HTTP. Which kinda makese sense. The schema gives you a great idea of what "should be", and the typing/errors/etc are understood by the host language.
You store your configuration as plain text in your repository and whatnot. When it comes to deployment you just compile it to a binary file.
Cap’n’proto also has plain text and JSON serialization formats if you really want to have your deployed config file be directly human-editable and deserialize from that. I was just noting a very cool feature of having your config written in cap’n’proto and it’s what Cloudflare uses to maintain a bunch of config internally if I read Kenton’s allusions to it correctly.
We had that a decade ago. It was called XML and XML Schema. All IDEs support it.
JSON was a huge step backwards in the name of simplicity. And now when we are going to add similar functionality to JSON, something else is going to come out in the name of simplicity (like NestedText).
Magento is a popular codebase that made XML-based configuration a fundamental part of its architecture. The results were terrible and caused numerous headaches and countless hours lost to trying to troubleshoot inscrutable configuration issues. The Magento 2 codebase began a shift away from XML for configuration, although it still uses some.
There may be room for an argument that Magento did XML badly (it did many things badly), but I don't believe I've ever seen XML done well.
I don't get it. The @Configuration and @Bean annotations are at least 100 times more readable and powerful than whatever garbage people used to write into their xml files to define beans. 20 lines of xml are often equivalent to like 8 lines of Java and each of those Java lines is shorter than the xml equivalent. Repeating closing tags is not very interesting.
If your child node has unique name among its siblings and does not contain nested nodes, then it's an attribute. Otherwise it's an element. Seems pretty obvious to me.
The fundamental issue with XML is its impedance mismatch with common data structures which forces using Object to XML mappers (whether explicitly or implicitly). It's more or less solved with XML Schemas or DTDs, but if you're looking at just XML, you can't tell whether some element is an array or a single node. Thus JSON is better suited for serialization.
> If your child node has unique name among its siblings and does not contain nested nodes, then it's an attribute. Otherwise it's an element. Seems pretty obvious to me.
That is really not what attributes are for. I feel a bit of a fraud posting that because I'm not an XML expert and so not really clear what they actually are for. (This reenforces the parent's point: you need to be an expert to know what such a fundamental feature is for.) I remember it's something like "something used to help interpret the actual value" e.g. units of measurement. But most of the time, even if it's non-repeating with no children, you're supposed to use elements rather than attributes.
One problem here is that attributes are so much more compact (and so often easier to read) than elements that it's tempting to use them in places where you ought to use an element (and many people over time have given in to that temptation). Another problem is that the distinction between attributes and elements is almost never useful. That was the parent comment's point by the looks of things.
> The fundamental issue with XML is its impedance mismatch with common data structures
That's probably part of it, but I think at least as problematic is that it has many features that most of the time you don't need and don't want to have to care about. Things like CDATA (also mentioned by the parent comment), custom entities, external entities, DTDs (which can be inline in XML files so you need to know all about DTDs to understand XML properly). That's why there are all sorts of weird XML vulnerabilities that JSON doesn't have. Did you know you can make an XML file that reads your /etc/passwd file when it's parsed? That is not an issue with JSON.
HTML tag and attribute is markup. Strip it and document would be still legible for a human being. Markup is non human part - presentation, semantic web.
Thanks, I found this explanation really helpful, and almost obvious in retrospect (as the best explanations often are!).
I had been thinking that all of these extra features that XML have are just a case of massive overengineering that no one would ever need. In fact it's a case of taking something fundamentally meant for text documents with extra markup, as the name implies, and misapplying it to config files and IPC messages which are just not the original domain at all.
Machine receives data, human receives application with documentation, builder. That's exactly what we have today except UI can be plugged to any stored document. To good to be true.
I think XML was killed by poor usability. Plain text XML, XHTML and XSLT authoring is not fun.
I am trying to uncover it from DOM perspective [1], so far I like it more than Markdown. XHTML and HTML is just a serialization format. HTML is not a good one [2], [3], [4]. XSLT may have nice GUI or compact syntax like RELAX NG.
> you need to be an expert to know what such a fundamental feature is for.
No you don't... the parent commenter explained to you what it's for in a simple and concise manner... you chose to not accept that even though you're not an expert in this, and then complains you need to be an expert to do it?!?
The parent commenter explained it in a somewhat obtuse way.
I don’t doubt they meant to be clear, but reading it they were not and raised more questions than were answered.
As an example:
Wouldnt attributes be better served as details about the current element?
Wouldn’t elements be better served as “I am a child of the parent”?
Why would I use an attribute as a “non-repeating child” when semantically that doesn’t make sense when looking at the document? The attribute is inside the element’s definition, and seems to me attributes should be used to further describe the element being presented itself, and not be structural or describe itself as a child in any way.
The parent commenter gave an explanation that, yes, was simple and concise, and also good enough for you to believe it (or you already thought that way). But it's also wrong. That just reinforces my point.
(The true difference is explained in sibling comments to yours, by sergeykish and tannhaeuser, if you're interested.)
> Did you know you can make an XML file that reads your /etc/passwd file when it's parsed?
Not only can SGML (but not XML on its own) read /etc/passwd, it can format it into fully-tagged markup and then render it into eg an HTML table. Demonstrating what SGML/XML is actually designed for: encoding and authoring semistructured text. This can't be overstated in discussions like these where use cases for config formats, service payload formats, and actual text authoring are all thrown into the same basket when they shouldn't.
Btw: you can parse and canonicalize this new config file format into markup using the same SGML mechanism you'd be using for CSVs like /etc/passwd, namely short references
Btw2: you can skip/ignore markup declarations in XML, including whole declaration sets (DTDs) since these can be recognized using plain greedy regexpes, though you can't ignore entity declarations when actually used in your XML body text
Exactly, jsonschema allows one to describe exactly how the json should look like including inter field validation. And with tools like reactjsonschemaform you can generate a ui on top of it for free.
XML is fatally flawed because you can't safely put one XML doc inside another one. Because of this rather fundamental problem, it never was any good for anything, and it never will be.
SOAP was and is an epic disaster, so that hardly seems like a refutation. The known way to embed an entire XML doc into a SOAP message was to use CDATA, which isn't a general solution because it means the embedded doc can't have ]]> in it anywhere. You could also base64-encoded the included doc.
Both of these solutions and all other known solutions to this problem are, as I'm sure you can see, just awful.
You can't just paste XML in XML because of the <?xml?> thing, because of entities, and because of half a dozen other misfeatures of XML.
Roughly speaking, you can do things like the following:
<!-- The special XMLNS attribute binds a short alias to a long name -->
<p:parent xmlns:p="urn:some:unique:string">
<c:child xmlns:c="urn:some:other:child:name" x=3 y=5>
<c:subchild> <!-- No need to repeat the fully qualified unique name -->
<p:tada>You can even interleave!</p:tada>
</c:subchild>
</c:child>
</p:parent>
Note that while this is possible to write by hand, typically namespaces are for documents generated and processed by tools. The XML Schema Definition (XSD) format has full support for namespaces, so you can define documents based on modular chunks. E.g.: you can "import" the SVG namespace into a diagramming XML document format namespace, but restrict its usage to only the child nodes of an "img" tag. Or MathML as the children of "graph" nodes. Both SVG and MathML can potentially import a shared "font" namespace. Or whatever.
In the XML Reader API, each element has a "fully qualified" name that includes the long namespace prefix. If you use the API correctly, your tool can handle nested documents, or gracefully ignore them if it's appropriate.
The fiddly part is making this efficient, i.e.: avoiding a full string comparison against a long URI or URN. You typically have to "register" the namespaces you're interested in, and the API gives you some sort of efficient token instead of a string to use from then on.
I'm not saying it's perfect. Nothing is in XML. It was designed by committee, it brought too much of the legacy SGML baggage with it, but its namespace capabilities are a lot better than nothing at all, in much the same way that C# or Java don't have perfect type systems, but they're superior to loosely typed languages.
You don't embed plain text XML in CDATA, right? You escape it
function escapeXml(unsafe) {
return unsafe.replace(/[<>&'"]/g, function (c) {
switch (c) {
case '<': return '<';
case '>': return '>';
case '&': return '&';
case '\'': return ''';
case '"': return '"';
}
});
}
Or you convert to the same encoding, strip XML declaration, expand entities. In short work with adequate tools.
Sure you can. At work we talk to a system that requires that we do exactly this. The solution they chose is entirely trivial and safe: include the embedded doc as a base64 encoded string...
> We had that a decade ago. It was called XML and XML Schema.
It would be true if XML was not full of all this SGML debris like "entities" (really, uncontroller macros), if real schema formats was flexible enough (I needed <c> inside <a> and <c> inside <b> when they totally different), etc.
But when a config reader tool has to deal with 40+-year legacy of enterprise guys wanting to embrace the universe, but all this doesn't allow to control contents without external measures like regexp checking... that simply shuts up facing real world.
I spent years working with xml, xslt, xml schema. Frankly when I first saw json I thought it was terrific. Nothing has changed my mind since. Why do you feel like it is a huge step backwards?
JSON Schema [1] is actually a mature standard now, with decent tooling support, mostly through OpenAPI (formerly Swagger), which extends it with support for endpoints.
It's much simpler to use than XML Schemas, and arguably results in cleaner data models, since it doesn't have anything analogous to XML namespaces that allow for arbitrary mixing of schemas.
I can see this work perfectly fine in typed languages like C#: `NestedText.Deserialize<T>("nestedtext")` where the deserialize method handles the actual mapping of nested text objects to `T` by providing the deserializer a class / classes that handles the string -> scalar(s) mapping for the given T. That would, sort of, function as a Schema.
I think the only thing, from glancing over the project, that would need to be supported to make this really useful is nested lists/dictionaries. I don't see how this can be done but maybe I'm missing it.
You can always do that, defining the schema in the client to produce sensible checks, even with JSON. The problem is that wherever the spec is underspecified is another place where two different clients can deserialize differently, and both be correct.
And the problem with stringly typed systems is that everything is underspecified
Came here to say the same, Cuelang is by far the best config system and paradigm I have tried. All else seems so last century, though Cuelang has its foundation in NLP systems from last century :]
Slightly off-topic, but yes, having fail-fast deserialisation is great.
I wrote a json/kotlin-serialisation library once and purposely restricted some json-features to achieve that:
1. Fields can arrive in any order - this is standard
2. Field names are matched case-insensitively - so keyA and keya are the same, because who would use two variables differing only by case. Serialization keeps the original casing of the name.
3. Missing fields throw an error. if they are nullable, they have to be explicitly set to null - so that you can be sure the serialization side upgraded to the latest version of a protocol if a field was added, and things don't just work by chance.
4. Nullable strings are not coerced to empty strings or anything like it. Kotlin is null-safe, so if it's a string, it has to be "". If it's, for whatever reason, a nullable string, you can set it to null.
5. Enums are also serialized case-insensitively - so you an write "keyA": "eNumVaLuE" if you want - typos should not break the code here, no on would you two enums differing only by case. IIRC booleans could also be TRUE, tRuE, truE etc. (but NOT t or f, or yes or no, or 0 or 1 or empty).
6. Superfluous properties are silently ignored.
These rules were a great tradeoff for quick development, mixing languages and having fail-fast behavior with a stable protocol.
> For example, in JSON 32 is an integer, 32.0 is the real version of 32, and “32” is the string version. These distinctions are not meaningful and can be confusing to non-programmers.
Au contraire mon frere. What's the point of this data format if it's only intended to be for humans and not computers, because for computers the data type is critically important. For example, I've more than once seen the extremely ill-advised idea to treat zip code as a numeric, which completely screws your data model once you want to support zip+4 or international postal codes that contain letters.
No, because you’re pushing the choice on the applications giving them more opportunities to screw up. If your data is already typed and something is a string odds are the application will either just follow that lead or look up why the data format authors picked that type in case it’s important or relevant.
I can't figure out what's happening here. Is everything just a string? They complain about how YAML handles booleans (fair criticism), however I can't see how NestedText fixes this.
This feels far inferior to TOML (everything NextText does, and much more) or JSON (very unambiguous, works natively in most languages).
I see this a lot in startups and open source projects. They exist solely as a criticism of their competition (which is fair!), but don't work to understand the real problems and fix them in any meaningful way. You can spot them when they talk a lot about what's wrong with the competition ("AWS is too expensive", "YAML is too ambiguous") but never explain their solution.
"It's a complex problem so we just cast everything to a sting and require applications to handle the typing" is what I interpret it as.
Honestly after a few years using YAML you remember to always quote anything you know you need to be a string, and the different multi line types.
An argument against this is "why not just design it to be simpler" — well, then it's less useful in as wide a variety of applications, and stands less chance of actually becoming a standard like JSON or YAML has. And you end up with the XKCD problem.
YAML came in and solved a few of JSON's most glaring problems (multi line and comments) with a usable approach.
This new format seems like it's "YAML, but a little different."
If I wanted to usurp YAML, I'd focus on the greatest pain points for most, whitespace and schema support.
Sadly my comment will not add value to the conversation, but I came to the comments looking to see if someone made the XKCD reference. To my delight, here you are. #927 always delivers - https://xkcd.com/927/
Interesting. Maybe the time is ripe for a next generation of config formats. It looks decent. But if you just open a file with this you might think it is YAML if there is nothing else to hint you?
I don't like any machine readable format that doesn't have some indicator that it is a complete document (like JSON or XML does). I've had a production issue where a format like this one was used where the file was read before it was finished writing it and ended up with a corrupt configuration as half of it was elided without any way to know.
This is the biggest problem with this format! You probably won’t need it for configs, but they are written by engineers which are fine with other formats. For business data missing list of dictionaries is a deal-breaker.
This is yet another shallow project similar to dribble-design portfolios of some people: looks cool at first glance, but has so big problems that it won’t fly.
This supposed "biggest problem" doesn't actually exist. Lists of dictionaries work fine, and I don't see any reason they wouldn't be useful for configs either. Maybe you should actually try it before dismissing it so shallowly?
This:
list of dicts:
-
key 1: Hi
key 2: there!
-
key 1: I'm a list
key 2: of dicts!
parses as
{'list of dicts': [{'key 1': 'Hi', 'key 2': 'there!'}, {'key 1': "I'm a list", 'key 2': 'of dicts!'}]}
Did you just guess that syntax? It's not in the documentation.
Hidden features effectively do not exist. The parent may have gone overboard with judgement, but this really is the fault of the project maintainers. I'll admit I mentally filed it in the "useless toy" category without this feature.
I went looking for it specifically because it's one of the ugliest and most confusing parts of YAML. To be pathological, what if you have a list of lists?
Sure, the documentation could be improved, but the syntax is extremely simple, consistent, and predictable. A list of lists is also exactly what you would expect:
list of lists:
-
- first sublist
- goes here
-
- second sublist
- is here
I don't think either of these are hidden. Yes, explicit examples would be nice, but assuming things not shown are impossible seems strange to me. Would you assume Python can't do lists of dicts, or lists of dicts of dicts? A quick search fails to find any examples on python.org (I did find lists of lists though).
I have no connection to this project. I'm speaking purely from my own experience of spending a few minutes looking at the docs and trying a couple of things in Python. The sublists and subdicts were obvious to me as soon as I learned the three forms lines can take. The only thing I wasn't sure about was whether the sublists or subdicts had to start on the next line, so I tried it out: yes they do.
What is the representation for objects in the “name and arguments form” (eg enums in rust, variants in Haskell, other tagged/discriminated unions)? It seems like they are becoming more popular in various programming languages.
What is the way to represent a map/dictionary where the keys are not strings? In json I also don’t have a good way except I guess a list of objects with a key and value field but it feels heavy. In Lisp I would probably use an alist (whereas a plist might be used for the “string” (or symbol, rather) keys). In fact, how does one even write a list of lists with this?
Interesting idea to remove number and string types. The downside being that you spend more time writing parsers of the data. But if you had some sort of separate schema that generated all of that for you, it would no longer be such a big deal.
They mention that their key/value pairs are ordered. The downside here is that not all languages (eg. javascript), support them.
I also prefer them to be unordered. The downside with ordered dictionaries, is that you need to always be asking "does sequence matter here?". So it adds an additional thing to think about, more tests need to be written, and certain optimisations can't always be made.
This looks fantastic. I was recently reading a config file from Rust and ended up going with JSON5 but this is simpler to read. In a language like Rust you don't need the types because you specify that in the struct anyways. Sure, this means that there are effectively more types than strings but the user doesn't need to differentiate.
So there is no need to worry about if maps are ordered or if a value is an integer, real or string in the format itself. Ironically for Python (which the reference implementation is in) it does seem much more annoying to have to manually call `int()` on each element.
I'm just a little sad that tabs are disallowed. I really think the best rule for indentation-sensitive languages is that each line must either have the same indentation in which case it is the same level, same indentation plus any amount in which case it is the next level, or the exact indentation of any previous level in which case it is a dedent. These "solutions" which just forbid tabs are half-assed and ones that try to convert tabs to a set amount of spaces just lead to confusion.
Additionally it would be nice if there was an example of a dict inside a list. I think it would work like the following but can't confirm from reading the site.
293 comments
[ 2.5 ms ] story [ 304 ms ] threadNo doubts that there are use cases for this, but calling something that casts everything to strings an alternative to the above formats seems like a bit of a stretch.
I'm really struggling with this assertion; IMHO one of the problems with JSON is the lack of more sophisticated scalar types.
That being said, this appears extremely readable, so my concerns could definitely be alleviated by a decent schema.
…only because NestedText does not support numeric types at all. That seems like throwing out the baby with the bathwater.
It's less of a problem if you're using it for configuration files, where a program knows what key's values need to be cast to an integer or float.
But it seems disastrous if you wanted to use it for storing or transmitting data, above all between applications. You're immediately throwing out the possibility of being able to serialize and then deserialize data in basically any programming language.
I shudder at the idea of an API that accepted NestedText, where I'd need to worry about whether my floating-point output was compatible with its floating-point string parser. Yikes. I want the serialization format to handle that. Isn't a major criticism of JSON that it doens't have a built-in datetime representation?
As for stringification, JSON's data types are mismatched with pretty much everything that isn't JavaScript to some degree. If you need to serialize a 64-bit integer to JSON, you serialize it as a string because the parser on the other end is probably going to try to parse it as a double-precision floating point number. Once you've started serializing numbers as strings anyway, it's not too far to "serialize every scalar as a string".
Indeed, not being able to have lists of dictionaries, or lists of lists, is very restrictive. Seems to be for very simple configurations only. E.g. a set of preferences, but not a set of monitor calibrations. (Inventing arbitrary dictionary keys seems pretty hacky.)
I don’t think this is true. None of the examples have lists containing non-string objects, but the documentation doesn’t seem to draw a distinction between lists and dictionaries wrt what can be placed in them.
(Both lists and dictionaries are initially described as only containing strings, and later this description is expanded to include nesting; this counterintuitive arrangement may explain the confusion.)
Frankly, in most languages, this is better because you don't have the types of objects randomly change based on user input. (In a few languages, with a few libraries, you can specify the type of the document to the parser and have it fail to parse the entire document if it can't deserialize to the right type, in which case this is a little weaker. But you can still do that with NestedText, just one step after the parser - have your own function that takes a ComplexStructure<..., String, ...> and returns a ComplexStructure<..., int, ...> or throws an error.)
Only in untyped or dynamically-typed languages. But even in JavaScript one may write +obj.version instead of obj.version to make it numeric. Evading this is a straight way to hell.
In a statically typed one, conversion is typically generated from description and type checking applies just at reading.
The problem with vaguely specified format is in more simplex cases. Shall we accept 45x as number (and what it value will be, 45 or 0)? 045? 045x? What date is 1/2/3, 1-2-3? And so on.
I think the point is that the answers to those questions may well be application-specific. In which case it is better to not bake them into the file format.
- There's no way to say that you want an integer; you get a floating-point value.
- See https://news.ycombinator.com/item?id=24676484 , you can't reliably accept integers over 2^53 without taking them as strings.
- Someone can always specify something of the actual wrong type. (Imagine changing YAML "version: 1.9.1" to "version: 1.10". You can't just stringify 1.10, you'll get "1.1"!)
So, in a practical data format, the schema for your document needs to say something like "This is a number, which must be an integer between 0 and 2^16" or "This is a string, make sure to quote it" or whatever, and a generic statically-typed JSON- or YAML-parsing library isn't going to handle that for you. And telling your users "the input format is JSON" doesn't answer that question: you must make it explicit to users.
Fortunately, you can handle it just fine in a statically-typed language in one of two ways. One is to accept an object from your parser that consists of variant types and pass it through your own function that validates it against a schema, and then either returns a more-restrictively-typed object or throws an error. Such a function could easily do string conversion too if given NestedText input, as I mentioned. The other is to pass some information into your parser saying, don't act like a generic JSON/YAML parser, instead interpret these particular fields in this particular way and accept only things with this structure. If you're doing that, you can easily tell the parser to use this particular string-to-integer function on the strings in NestedText and then return an appropriately-typed object containing an integer to you.
One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
There are also issues when something looks numeric but doesn’t parse (eg 1.2.3, 3/2, 12in, 4h30m2s, 2:30, 2020-02-29, etc). One way to deal with these is a tokenisation rule like in Common Lisp: if it is a valid number syntax then treat it as a number, otherwise it’s a symbol, but this can lead to issues (eg you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes. It seems crazy to pass that detail on to the poor sod who has to write the config file).
The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way.
Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false".
The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.
Unless you have actual type annotations/tags (eg xml, jsonld, graphql), everything IS a string. There's no assumptions otherwise.
http://p3rl.org/guts#Magic-Variables
Perl tied-variable magic just means there are (effectively) getter and setter properties attached to the variable. "Magic" is just the name that was chosen in the implementation, and it stuck.
It's used to implement variables with special, automatic meanings, like $$ for "current pid" and $! for "last error".
It's also used to implement variables with user-defined behaviours on access, which is quite handy for a lot of abstractions.
A lot of modern languages support both of these things, because they are useful, but it's not called magic in those languages, it's called something like "watchers", "proxies", "getters and setters" or "hooks".
No, the criticism of YAML-style "magic" is that it leads to entirely surprising behaviour from innocuous input. Perl magic is not that kind. If you're using a special variable, you already know why.
In fact in Perl, you can opt for longer, readable, lexicon over the terse single character variables; and that’s literally how modern Perl should be written.
Whereas the problems described with YAML is where it can automatically alter your data based on what the parser “thinks” the data should represent. Which is generally what people mean when they talk about “magic” in IT: systems that don’t honour your input and instead automatically convert it into something else. Perl doesn’t do this even in spite of it looking like executable line noise to many.
If you take a look at how string handling works in most programming languages there is a lot of "magic" going on there. Which isn't a bad thing necessarily, because most programmers don't want to deal with the intricates of strings unless they really have to. The key is that this magic doesn't get in your way and doesn't do too magical things nobody ever asked of it.
https://yaml.org/spec/1.2/spec.html#id2805071
Of course, even though YAML 1.2 is a decade old, there are still many parsers that accept YAML 1.1.
https://yaml.org/spec/1.2/spec.html#id2781553
https://yaml.org/spec/1.1/#id895631
YAML has actual type annotations (tags).
I've never seen typed yaml, this is wild.
Can't say I love the notation, but indeed that is type annotations. I guess neither "yaml type hints" nor "yaml type annotations" are the right query. Had to search explicitly for "yaml tags".https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
In Python:
In my browser's JavaScript console: (Consider what happens if you try to send a tweet's ID, a perfectly normal number like 205052027259195393, through JSON. Or if you try to serialize a stack trace on a 64-bit system, where addresses are also perfectly normal numbers.)Javascript number is a floating point
no automatic promotion to BigIntThe JSON standard doesn't place restrictions on size or precision of numbers, instead just noting that implementations can vary their treatment of and limits on numbers. While JS uses doubles for all numbers, many other languages emit an integer type for a JSON integer. So, once you go beyond the range where a double can accurately represent all integers, you run the risk of a mismatch in how the number is interpreted by different languages parsing the same JSON.
Of course the spec also allows you to create way too big or too precise numbers that would be problematic in most languages as well; it's just that this is a somewhat common bugbear.
I wouldn't necessarily call it a flaw in JSON though, more an issue with JSON.parse or really just a fact of life when dealing with numbers in JS. Alternatives to the built in JSON.parse exist to read large integers as strings or bigints.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Described issue is not problem of a JSON but engine which parsed it and language which stands behind the parser. Any config format will eventually have same result and same issue.
So forcing programmer to parse every single piece of data for sake of "it's his responsibility" is not a case here.
I also disagree this is in any way programmer responsibility to create standardized way of creating parser for everything. This format gives you nothing but indentation so you are forced to create documentation for everything field, what type it's and what kind of values it takes. Lots of extra work for nothing when you have any other format.
I mean, there's a pretty clear argument here: Twitter themselves used to return these numbers as numbers in the API until they realized they were about to hit this problem. https://developer.twitter.com/en/docs/twitter-ids
That's an incredibly weird complaint when the real problem is that javascript's JSON.parse doesn't use BigInt for large numbers.
I don't think that's what this tool is for. This tool is for humans to read and edit data. That's a different use case from automated programs exchanging data, for which I agree you should be using standardized numeric and boolean data types and not making everything a string. But how that standardized data gets determined from data that humans enter should be up to the individual application.
> A key that requires quoting must not contain both single and double quote characters.
You can't really serialize user data with that restriction.
Maybe we don't need it, but it often helps.
Also, this data format isn't necessarily for humans to exchange data with other humans, but for humans to give data to applications in a format that's much easier for humans to use.
> if this is insufficient to communicate it to the machine
Not at all. Each specific application can easily parse this data format according to its own needs. What this data format doesn't specify is a single translation into application data that is the same for all applications. But applications don't need or want that, because they have different use cases.
If you make everything a string then the interpretation of "no" as boolean true or false is left to each tool, and there are even tools which have different interpretations of "yes/no" for each field.
Most likely if their input is meant to be machine interpreted they would need to be trained to provide specific inputs anyway. I like that NestedText doesn't hide that problem. It lets the user organization decide how it wants to manage that problem, and what symbols or words are understood by the people authoring the files.
ISO8601 joins the chat.
The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined.
The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.
The spec requires that a full 64 bits of signed integer be parsed and understood, that hexadecimal, octal, and binary values be integers, and that floating point values be parsed as doubles.
It doesn't support hexadecimal float, however, which is a pity: having a guaranteed bit-identical format is a nice affordance.
Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot.
> you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes
Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that)
(disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda)
[1] http://amzn.github.io/ion-docs/
Only if this format is intended for use-cases that never need to deal with numbers.
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
That statement couldn't possibly be more wrong.
Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following:
- Unsigned 64-bit numbers.
- A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error?
- NaN
- Infinity
- Negative zero
- Denormal numbers.
- Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats!
- Efficiently encoding floating point to use the minimum digits without losing precision.
- Parsing those minimal numbers with perfect "round-tripping".
- Doing the above efficiently.
- Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser...
The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient.
Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that.
Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit!
The difficult bit is the parsing.
1. Declare numbers as numbers in the configuration language. E.g. "decimal(1e1000)".
2. Parse declared numbers with a lossless format like Python's decimal.Decimal.
3. Let users decide at their own risk if they want to convert to a lossy format like float.
That doesn't matter at all. The author's aims will be ignored if this format is used for anything even vaguely important. Eventually it'll need tooling to both read and write it.
DevOps pipelines, applications with GUIs, or something will need to both parse and generate this format in a consistent way.
There is no such thing as a human-write-only format in widespread use.
Even programming languages are regularly generated by tools such as RPC API codegen tools, LINQ-to-SQL and the like.
One example you provide is decimals for currency values but I claim you would want such values to look like $1234 in config files so that when they are reviewed or written, the person reading the file knows they are looking at a dollar value and can be concerned if it is too large.
I’m not suggesting that applications write their own number parsing. Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.
> Just do uint64::parse or parseInt or Double.of_string, or whatever else you need to access your language’s number parsing routines.
Okay, so the computer is doing the parsing.
Those functions are notoriously inconsistent in their behaviour, particularly across different programming languages. If you're not careful, you'll end up accidentally using the internationalised versions of those functions. Even if you're careful, other people won't be.
Remember, data formats are for interchange. They have to be language agnostic. They have to be well-defined, and it should be possible to write a parser for them without having to guess at the precise details.
The harmful consequences of the Robustness Principle are now well-recognised in computer science: https://tools.ietf.org/id/draft-thomson-postel-was-wrong-03....
Some things need to be done properly, nor not at all.
I am far more worried about localisation issues than language issues. If you are storing something central to multiple applications I'd argue a text file is the wrong tool
And - it is certainly OK in many instances to have fixed-width, fixed byte-order binary encoding as the format's basis. It comes with the twin downsides of wholly different categories of errors cropping up, and with the lack of a universally agreed upon tool for human entry.
Perhaps text was a fashion, though. I definitely have had thoughts in that vein lately. And in that case we shouldn't always be rushing to use it as the source of truth when we have many good, machine-level agreements about numeric formats.
It seems you're mixing up the language definition with implementations that try to follow the language definition.
If different implementations have different results then either they are buggy or the language has some important holes in the specification.
Either way,the solution to this problem is not less validation.
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”.
That's the whole point, isn't it?
I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions?
Wouldn't it simply easier to let the language and the parser do that already?
I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.
Edit: A lot of people seem to be accidentally clicking the downvote icon.
Your editor adds the whitespace to match the current scope, so you don't have to type it manually.
What's your issue with significant whitespace?
I dislike significant indentation for configuration because the nesting tends to get deeper than for programming languages and it can be harder to see what's going on.
Modern languages repair this with mandatory block braces: {C style} (Rust, Go, Swift...) or Modula/Ada style (if x then foo else bar end). C style is easier for editors.
> What's your issue with significant whitespace?
1. Many cases when leading whitespace is still spoiled, including editors (which use can't be avoided due to corporate tool limiting), web formatters, etc. Last decade spreading of mobile-aware tools made this worse.
2. With grouping by indentation, it's hard to determine block start if multiple blocks are ended at the same line, like:
and let's imagine the most nested block occupies 2-3 screens (not rare for configs or code, even if it is against good coding rules). You have no means to decide what block start is to be matched when you ask editor to find match.I code in Python continuously since 2004 and I deem grouping by indentation is bad idea. Not fatally bad, of course, but with explicit grouping it would be a bit better.
That assumption is... not applicable to most scenarios I come across and will likely lead to issues being pushed downstream and introduction of subtle bugs and defects.
> I once disabled our product for the entire country of Norway for a day because `NO` in YAML evaluates to `false`
https://twitter.com/aarondjents/status/1307692593493553160
There should be at least some support for some standardized representations (basically JSON + ISO8601 datetime + some encoding for embedding whatever stringified serialization, eg. just how HTTP uses chunks and unique boundary tokens).
One might call this 'argument by gotcha' interesting to imagine the kind of lifeworld that makes one think this is a worthwhile form of persuasion.
In Emacs, org files are often used for configuration similarly.
While I'm a fan of EDN most of the time, which doesn't suffer from the issues they mention JSON having, not having to wrap things in quotes for text, and using indent for nesting definitely would make it nicer for non technical people. So I can see a use case for this as a simple text interface for forms that is targetted at non-technical users.
I want to know beforehand what I can put in a config file and I want a fast and hard failure if what I put in there is not good.
And this should be implemented at the file format parser level, with hooks for apps to add on top of the default behavior, so that every app that implements this format gets these things almost for free.
It would be nice if we had such tools.
It looks oddly like HCL. I wonder...
I know cap’n’proto also has fantastic support for using the schema for config files. You can just compile any constant as a stand-alone serialized message that you mmap into your code in a safe way. It can’t do complex math and things (at least yet) but you can express lists, dictionaries, and reference other constants, so as a config file replacement I love it. I’ve also found the format to be far more regular and consistent than you get with things like text protobuf (you’re still using the schema language instead of another format)
Are you suggesting using a binary format for your config files? I think most people would find that more trouble than a decent text format.
> ... than you get with things like text protobuf
You can just use protobuf's canonical JSON representation (thought the lack of ability to use comments is annoying).
Cap’n’proto also has plain text and JSON serialization formats if you really want to have your deployed config file be directly human-editable and deserialize from that. I was just noting a very cool feature of having your config written in cap’n’proto and it’s what Cloudflare uses to maintain a bunch of config internally if I read Kenton’s allusions to it correctly.
You can then compile it into whatever format (JSON, plain text, binary) that you want for actually reading it from disk.
[0] https://www.schemastore.org/json/
JSON was a huge step backwards in the name of simplicity. And now when we are going to add similar functionality to JSON, something else is going to come out in the name of simplicity (like NestedText).
There may be room for an argument that Magento did XML badly (it did many things badly), but I don't believe I've ever seen XML done well.
In a minute I can read and write json from most languages I use.
In the same amount of time, I'm still wondering if I should use a tag or attribute in xml. cdata? expat?
It's not that xml isn't a good technology. It's that it's not appropriate for general use, especially in comparison to simpler alternatives.
The fundamental issue with XML is its impedance mismatch with common data structures which forces using Object to XML mappers (whether explicitly or implicitly). It's more or less solved with XML Schemas or DTDs, but if you're looking at just XML, you can't tell whether some element is an array or a single node. Thus JSON is better suited for serialization.
That is really not what attributes are for. I feel a bit of a fraud posting that because I'm not an XML expert and so not really clear what they actually are for. (This reenforces the parent's point: you need to be an expert to know what such a fundamental feature is for.) I remember it's something like "something used to help interpret the actual value" e.g. units of measurement. But most of the time, even if it's non-repeating with no children, you're supposed to use elements rather than attributes.
One problem here is that attributes are so much more compact (and so often easier to read) than elements that it's tempting to use them in places where you ought to use an element (and many people over time have given in to that temptation). Another problem is that the distinction between attributes and elements is almost never useful. That was the parent comment's point by the looks of things.
> The fundamental issue with XML is its impedance mismatch with common data structures
That's probably part of it, but I think at least as problematic is that it has many features that most of the time you don't need and don't want to have to care about. Things like CDATA (also mentioned by the parent comment), custom entities, external entities, DTDs (which can be inline in XML files so you need to know all about DTDs to understand XML properly). That's why there are all sorts of weird XML vulnerabilities that JSON doesn't have. Did you know you can make an XML file that reads your /etc/passwd file when it's parsed? That is not an issue with JSON.
Confusion arise once a human observer is lost.
I had been thinking that all of these extra features that XML have are just a case of massive overengineering that no one would ever need. In fact it's a case of taking something fundamentally meant for text documents with extra markup, as the name implies, and misapplying it to config files and IPC messages which are just not the original domain at all.
I think we should draw on XML strength points. People read articles in browser, not plain text. "Add to cart" is just a POST request with id
yet we have forms and interactivity. Like in literate programming text and data live together, interactive application like a Smalltalk image.In XML we can separate data from presentation.
Machine receives data, human receives application with documentation, builder. That's exactly what we have today except UI can be plugged to any stored document. To good to be true.I think XML was killed by poor usability. Plain text XML, XHTML and XSLT authoring is not fun.
I am trying to uncover it from DOM perspective [1], so far I like it more than Markdown. XHTML and HTML is just a serialization format. HTML is not a good one [2], [3], [4]. XSLT may have nice GUI or compact syntax like RELAX NG.
[1] http://sergeykish.com/live-pages
[2] http://sergeykish.com/script-style-is-cdata-in-html
[3] http://sergeykish.com/pre-newline-ignored-in-html-test
[4] http://sergeykish.com/content-after-html-appended-to-body-in...
No you don't... the parent commenter explained to you what it's for in a simple and concise manner... you chose to not accept that even though you're not an expert in this, and then complains you need to be an expert to do it?!?
I don’t doubt they meant to be clear, but reading it they were not and raised more questions than were answered.
As an example:
Wouldnt attributes be better served as details about the current element?
Wouldn’t elements be better served as “I am a child of the parent”?
Why would I use an attribute as a “non-repeating child” when semantically that doesn’t make sense when looking at the document? The attribute is inside the element’s definition, and seems to me attributes should be used to further describe the element being presented itself, and not be structural or describe itself as a child in any way.
(The true difference is explained in sibling comments to yours, by sergeykish and tannhaeuser, if you're interested.)
Not only can SGML (but not XML on its own) read /etc/passwd, it can format it into fully-tagged markup and then render it into eg an HTML table. Demonstrating what SGML/XML is actually designed for: encoding and authoring semistructured text. This can't be overstated in discussions like these where use cases for config formats, service payload formats, and actual text authoring are all thrown into the same basket when they shouldn't.
Btw: you can parse and canonicalize this new config file format into markup using the same SGML mechanism you'd be using for CSVs like /etc/passwd, namely short references
Btw2: you can skip/ignore markup declarations in XML, including whole declaration sets (DTDs) since these can be recognized using plain greedy regexpes, though you can't ignore entity declarations when actually used in your XML body text
Both of these solutions and all other known solutions to this problem are, as I'm sure you can see, just awful.
You can't just paste XML in XML because of the <?xml?> thing, because of entities, and because of half a dozen other misfeatures of XML.
You put XML fragments inside a parent XML document using namespaces.
This is very well supported, and used extensively.
Trying to "escape" XML to nest it in a parent XML document is Wrong with a capital W.
could you post or link to an example? i'm not very familiar with advanced XML features
or for a simple example: what would it look like to put `child` into `parent` using namespaces?
In the XML Reader API, each element has a "fully qualified" name that includes the long namespace prefix. If you use the API correctly, your tool can handle nested documents, or gracefully ignore them if it's appropriate.
The fiddly part is making this efficient, i.e.: avoiding a full string comparison against a long URI or URN. You typically have to "register" the namespaces you're interested in, and the API gives you some sort of efficient token instead of a string to use from then on.
I'm not saying it's perfect. Nothing is in XML. It was designed by committee, it brought too much of the legacy SGML baggage with it, but its namespace capabilities are a lot better than nothing at all, in much the same way that C# or Java don't have perfect type systems, but they're superior to loosely typed languages.
And yes, I'm being sarcastic.
Well, except for handling complex content documents like in all ebooks and, in sgml form, all webpages like this one.
It would be true if XML was not full of all this SGML debris like "entities" (really, uncontroller macros), if real schema formats was flexible enough (I needed <c> inside <a> and <c> inside <b> when they totally different), etc.
But when a config reader tool has to deal with 40+-year legacy of enterprise guys wanting to embrace the universe, but all this doesn't allow to control contents without external measures like regexp checking... that simply shuts up facing real world.
It's much simpler to use than XML Schemas, and arguably results in cleaner data models, since it doesn't have anything analogous to XML namespaces that allow for arbitrary mixing of schemas.
[1] https://json-schema.org/
I can see this work perfectly fine in typed languages like C#: `NestedText.Deserialize<T>("nestedtext")` where the deserialize method handles the actual mapping of nested text objects to `T` by providing the deserializer a class / classes that handles the string -> scalar(s) mapping for the given T. That would, sort of, function as a Schema.
I think the only thing, from glancing over the project, that would need to be supported to make this really useful is nested lists/dictionaries. I don't see how this can be done but maybe I'm missing it.
And the problem with stringly typed systems is that everything is underspecified
I should have mentioned that I want something simple and readable.
I wrote a json/kotlin-serialisation library once and purposely restricted some json-features to achieve that:
1. Fields can arrive in any order - this is standard
2. Field names are matched case-insensitively - so keyA and keya are the same, because who would use two variables differing only by case. Serialization keeps the original casing of the name.
3. Missing fields throw an error. if they are nullable, they have to be explicitly set to null - so that you can be sure the serialization side upgraded to the latest version of a protocol if a field was added, and things don't just work by chance.
4. Nullable strings are not coerced to empty strings or anything like it. Kotlin is null-safe, so if it's a string, it has to be "". If it's, for whatever reason, a nullable string, you can set it to null.
5. Enums are also serialized case-insensitively - so you an write "keyA": "eNumVaLuE" if you want - typos should not break the code here, no on would you two enums differing only by case. IIRC booleans could also be TRUE, tRuE, truE etc. (but NOT t or f, or yes or no, or 0 or 1 or empty).
6. Superfluous properties are silently ignored.
These rules were a great tradeoff for quick development, mixing languages and having fail-fast behavior with a stable protocol.
(https://medium.com/@fabianzeindl/generated-json-serialisatio...)
Au contraire mon frere. What's the point of this data format if it's only intended to be for humans and not computers, because for computers the data type is critically important. For example, I've more than once seen the extremely ill-advised idea to treat zip code as a numeric, which completely screws your data model once you want to support zip+4 or international postal codes that contain letters.
I can't figure out what's happening here. Is everything just a string? They complain about how YAML handles booleans (fair criticism), however I can't see how NestedText fixes this.
This feels far inferior to TOML (everything NextText does, and much more) or JSON (very unambiguous, works natively in most languages).
I see this a lot in startups and open source projects. They exist solely as a criticism of their competition (which is fair!), but don't work to understand the real problems and fix them in any meaningful way. You can spot them when they talk a lot about what's wrong with the competition ("AWS is too expensive", "YAML is too ambiguous") but never explain their solution.
Honestly after a few years using YAML you remember to always quote anything you know you need to be a string, and the different multi line types.
An argument against this is "why not just design it to be simpler" — well, then it's less useful in as wide a variety of applications, and stands less chance of actually becoming a standard like JSON or YAML has. And you end up with the XKCD problem.
YAML came in and solved a few of JSON's most glaring problems (multi line and comments) with a usable approach.
This new format seems like it's "YAML, but a little different."
If I wanted to usurp YAML, I'd focus on the greatest pain points for most, whitespace and schema support.
AFAIK they were both independently created in 2001 and YAML was not created in response to JSON.
a) Do not have values of different types occupy the same fields
b) Have a schema defined (explicitly or implicitly as part of the parsing), especially because you're likely working with a type system
This isn't good if you want `x = parseJSON(blob)` kind of API, but that's definitely not what you want for any kind of human-editable config.
It seems simpler than TOML, I'd give it a try.
It'd be easy to employ ... as a document separator / end indicator that could be checked for.
Taking the main example, what do you do if you have more than one vice president?
This:
parses asHidden features effectively do not exist. The parent may have gone overboard with judgement, but this really is the fault of the project maintainers. I'll admit I mentally filed it in the "useless toy" category without this feature.
I went looking for it specifically because it's one of the ugliest and most confusing parts of YAML. To be pathological, what if you have a list of lists?
It just shifts problems to other areas, which some may find even worse.
What is the way to represent a map/dictionary where the keys are not strings? In json I also don’t have a good way except I guess a list of objects with a key and value field but it feels heavy. In Lisp I would probably use an alist (whereas a plist might be used for the “string” (or symbol, rather) keys). In fact, how does one even write a list of lists with this?
They mention that their key/value pairs are ordered. The downside here is that not all languages (eg. javascript), support them.
I also prefer them to be unordered. The downside with ordered dictionaries, is that you need to always be asking "does sequence matter here?". So it adds an additional thing to think about, more tests need to be written, and certain optimisations can't always be made.
In Rust you do something like this:
So there is no need to worry about if maps are ordered or if a value is an integer, real or string in the format itself. Ironically for Python (which the reference implementation is in) it does seem much more annoying to have to manually call `int()` on each element.I'm just a little sad that tabs are disallowed. I really think the best rule for indentation-sensitive languages is that each line must either have the same indentation in which case it is the same level, same indentation plus any amount in which case it is the next level, or the exact indentation of any previous level in which case it is a dedent. These "solutions" which just forbid tabs are half-assed and ones that try to convert tabs to a set amount of spaces just lead to confusion.
Additionally it would be nice if there was an example of a dict inside a list. I think it would work like the following but can't confirm from reading the site.