I'm glad to see people experimenting with alternative document/object representations, but this one might be a hard sell: based on the README[1], it only has Python, Zig and Janet implementations so far. One of the nice things about YAML (and JSON, TOML, etc.) is that they have decently mature C, C++, or Rust libraries that other languages bind to.
But I find it incredibly annoying to estimate indentation when lines are wrapped in an editor (or webpage). Or, to a lesser extent but still throws me off, when multiple blocks end at the same line. Or when pasting blocks into another block, and having to double check to make sure the indentation was carried over correctly. I like editors that visually show indentation characters.
It really doesn't matter. Just force the leading indent to be exactly the same bytes. If indent moves between two values where one isn't a prefix of the other raise an error.
No it isn't. That is the point. If you have one line indented with 4 spaces and one line indented with a tab there is no correct answer what the difference of indent is. The only good option is to raise an error.
Looks confusing to who? Another coder? Then they have issues.
I get the original was talking about client facing config files. I'd rather see INI style config files personally.
If you're writing ugly code, braces or spaces won't save you. Just don't write ugly code.* Write it like the next person to view your code is an axe murder that knows where you live, so don't make them mad. You can minify later.
Visual clutter, familiarity to non-coders. Curly braces are almost never used outside of programming and are ugly to boot.
My benchmark for yaml/JSON alternatives is "how would I feel explaining it to a busy, sceptical client?"
If the intended audience is purely developers, then sure. JSON (with the addition of comments and trailing commas) is just fine.
White space has the additional advantage of agreeing with itself. Other demarcations can have issues where the indentation and the structure contradict each other.
I haven't found "this needs to be indented exactly the right amount or it won't work" to be much easier for non-programmers than "this needs to be enclosed in braces or it won't work." Most people have at least experienced parentheses in math (albeit maybe decades ago), so it's not an entirely foreign concept. Either one requires a bit of learning, but I think most people are capable of it, so any improvement in non-coder familiarity seems minor at best, vs. the very real costs.
Counter-argument - why do programmers insist on clear indentation if it doesn't aid readability? The indentation is there for humans and the braces are there for the compiler.
Again the dreaded Cobol argument. We had to struggle with a lot of this in the past: Cobol, SQL, YAML, BDD. All this would be much easier without this nonsensical idea that nontechnical people will read code. They won’t. Making code a bit more like prose doesn’t make it readable for nontechnical people. Yet we again and again make our life harder - ugly syntax rules, no code completion, no auto-formatters.
Please stop making code easy for non-coders. They don’t want to read it. They never did. They just want this damn box to work.
as a counter argument. I work in robotics, where many operators will look at and change settings in a yaml file during testing. They do not have software skills outside of this.
My educated guess then is you could have gotten them to change "settings" in C, Java or basically anything.
Just put the file in the root folder, and keep it as simple as possible and you should be fine? I mean, if they manage to write yaml correctly and consistently C is no match?
The reason the situation is the way it is now is precisely because the code being made easy for non-coders increased the popularity and reach of the products. Probably because non-coders also found it easy to pick up and start working with it.
That's just display though, if you have to show it to a skeptical client, why not run it through a browser that shows it without braces? It's the same as showing a webpage instead of the html.
YAML is a superset of JSON. In other words: any syntactically valid JSON file is a valid YAML file. If you want braces like JSOM, but not quoted strings, YAML supports it.
If the HomeAssistant subreddit is anything to go by its their biggest complaint (HA configuration is in YAML).
With that said if they weren't complaining about white space they'd be complainit about missing semicolons, missing/extra commas, missig equals signs, missing closing )]} or whatever.
Are you arguing "braces are easier to get right than indentation" or is this a point specific to YAML's rules? Because I'm not defending the latter but I find it hard to understand I would need to argue against the former.
I'm not buying that you genuinely have a target audience of "I trust this person with config files but their eyes are too gentle to see a curly brace".
Well. I've genuinely had clients editing YAML so there's that.
I can definitely think of a broad range of people where I'd be happy to recommend they use text files for config and data but I wouldn't be happy if those text files needed to follow the rules of JSON syntax.
I mean - to some extent I would rather not edit JSON. It's not a terribly ergonomic experience. If I had to design a format for my own use, it would be indentation based and probably look a little bit like YAML, Markdown or similar.
My google ability didn't see anything, but are their any case studies that show it's more readable? I'm happy to accept that it is, but I can't help wondering if research has been done or it's mostly gut feeling / anecdotes / aesthetics.
I'm not clear exactly what the "it" here refers to but as I mentioned in other comments it's fairly self-evident that indentation is easier to visually parse than braces. A simple thought experiment - would you find it easier to skim read code where the indentation was consistent with the bracing or where it was inconsistent? Your brain registers the indentation first and you only resort to counting braces if there's a reason to doubt the former.
Reality often times runs counter to our expectations though, which is why I wondered if no brace methods have been shown to enable simpler to understand usage / understanding.
>My benchmark for yaml/JSON alternatives is "how would I feel explaining it to a busy, sceptical client?"
My benchmark is this: can an autoformatter do its job every time without breaking something that's technically working right now but possibly formatted wrong?
Every data format that cannot comply with this contains in it a huge waste of time. Even as a python programmer, I extend the same rule to programming languages.
I use it to generate the config files, and pass the generated config to the application the same way you’d pass any other json/yaml/etc to your app. Neither ever talk to each other, and Dhall doesn’t exist within our runtime environment.
Huh, no. YAML is a superset of JSON. So valid JSON is valid YAML. This is sometimes suprisingly useful. Also, YAML is used everywhere and like the other user pointed out, has mature well tested libs for almost every language.
Any valid json can be parsed by any 1.2 yaml parser. That's how it gained popularity in the first place: you didn't have to migrate. Like ascii and utf8.
While true, this is what being a supperset means. What it means is that any JSON document is, without modification, valid YAML document and can be read with a YAML parser.
YAML supports tagged nodes and multiple documents in a stream, which can’t be represented in standard JSON. (You could make up conventions, but only your parser would support them.)
It just is: Any valid JSON is valid YAML with the same semantics. It was intentionally designed that way (though not from the start iirc).
Basically take JSON, make quotes optional for strings and make curly brackets optional if the object is indented properly, and boom you've got (something like) YAML.
My opinion only: I love JSON because it lacks so many foot guns of yaml. If you’re doing lots of clever stuff with yaml you probably want a scripting language instead. Django using Python for configs made me fall in love with this. Spending years with the unmitigated disaster that is ROS xml launchfiles and rosparams makes me love it even more.
Yaml and toml are fine if you keep it simple. JSON direly needs comments support (but of course wasn’t designed to be used as a human config file format so that’s kind of on us). And not just “Jsonc that sometimes might work in places.”
Beyond that, I think we generally have all the things we need and I don’t personally think we need yet another yaml. =)
> Django using Python for configs made me fall in love with this.
I also started advocating in-language configuration files (Python for Python, but also Lua for Lua, etc) a number of years ago because it lets you do really useful things (like functionally generating values, importing shared subsets of data, storing executable references, and ensuring that two keys return the same values without manual copy/paste) all without needing to spec and use Yet Another Thing™ that does only a fraction of what the programming language you're already using already does.
That also implies that you can't just test a foreign config file without first reading and understanding what it does, as just using one would imply arbitrary code execution.
This is a place where Tcl excels. You can easily create restricted sub-interpreters that can't do anything dangerous. If you need more power for trusted scripts you just reenable selected commands.
These aren't foot-guns per se, but I can think of another handful of grievances I have with JSON:
* JSON streaming is a bit of a mess. You can either do JSONL, or keep the entire document in memory at once. I usually end up going with JSONL.
* JSON itself doesn't permit trailing commas. I can measure the amount of time that I've wasted re-opening JSON files after accidentally adding a comma in days, not hours.
* JSON has weakly specified numbers. The specification itself defines the number type symbolically, as (essentially) `[0-9]+`. It's consequently possible (and common) for different parsers to behave differently on large numbers. YAML also, unfortunately, has this problem.
* Similarly: JSON doesn't clearly specify how parsers should behave in the presence of duplicate keys. More opportunity for confusion and bugs.
the funny thing is that json doesn't even need commas, they essentially act as whitespace, any amount or no amount would make no difference in the meaning of the document.
Running prettier (https://prettier.io) on each save will fix trailing commas for you. If you accidentally have one, it will just sneakily remove it and turn your document into one that is valid.
And the flip side of that with YAML is you can stream it, but you don't know once you've gotten to the end if it was the whole document without some user defined checksum mechanism.
Ran into a great bug with the INI format which has the same issue. The application would read the config file on modification but if you just wrote over the file it would sometimes read the config before the file was fully written. Have to use a temp file and move it rather than just edit it.
I believe that's only true if one were to load YAML via the "SAX"-style per-event stream, and not the "object materialization" that normal apps use (aka `yaml.load_all` or JAX-B objects) since in those more data-object centric views, where would one put the processing events for those markers?
I also originally expected `yaml.parse(...)` to eat them as it does for comments and extraneous whitespace, but no, it does in fact return dedicated stream events for them, so TIL
It may have been a good or bad decision. But comments were intentionally left out of JSON to avoid obvious ways to sneak in parsing directives and thus incompatibilities between different JSON-parsers.
If I had a penny every time someone tried to parse xml using a regex, if that classifies as a parser. Those are 100% incompatible with everything else.
Easiest way to demonstrate how wrong that is, is to throw in a comment in the example document ;)
JSON5 is the way to go. It supports comments and trailing commas. Unfortunately it's going to be difficult to supplant legacy JSON, which is so pervasive.
Except parsing JSON5 in browser is super slow. Native JSON.Parse doesn't support it, non-native parsnips are slow, and the only fast way to parse it is `eval()`.
The desire to use a single interchange format for all data is the problem. There are plenty of reasons to support comments and minor syntax issues that JSON itself dislikes for human consumable and interactive JSON. I'd think software JSON could be just that.
The introduction keeps citing "no need for escaping or quoting" as a major advantage, but provides no examples of what a key with a colon, or value beginning with "[", or any datum with leading or trailing whitespace would look like.
Also, the changelog is quite frightening!
> [In 3.0], `[ ]` now represents a list that contains an empty string, whereas previously it represented an empty list.
This made my curious to find out. The "Language introduction" docs [1] answers these points;
* keys containing leading spaces, newlines or colons can be represented with the multiline key form, where each line of the key starts with `: `.
* leading or trailing space is not complicated; the string values are just the rest of the line after the separator from the key, `: `. The values are not trimmed.
* a string value beginning with `[` just works in most places. This would not be confused with list values, as these would only start after a new line. Only in the compact inline list and inline dict form there are restricted characters for syntax.
It seems that their claim, no escaping required, holds. The slightly more verbose form of the language constructs may be required to represent special values though.
It’s a deterministic subset of Python. This means that if you have complex or repetitive configurations, you can use loops and functions to structure them. But it’s impossible to write an infinite loop or recursion.
Interesting! I started using jsonnet this year, but found that the language was needlessly quirky (e.g. the `::`, purely functional aspect, and no one wants to learn a new language to write configuration in the first place). More importantly, it is extremely slow (lazy evaluation without memoization...): rendering the Kubernetes YAML of my 5-container app taking over 10 seconds...
> It’s a deterministic subset of Python. This means that if you have complex or repetitive configurations, you can use loops and functions to structure them. But it’s impossible to write an infinite loop or recursion.
Starlark is indeed deterministic and guaranteed to terminate (the Go implementation has a flag that allows recursion, but it's off by default), but these are two orthogonal properties.
> Starlark is a dialect of Python. Like Python, it is a dynamically typed language with high-level data types, first-class functions with lexical scope, and garbage collection.
If it has first-class functions, how can you avoid infinite recursion? Like, what stops me from running the omega combinator in it? This is why Meson (a similar language) does not allow those kinds of shenanigans, to keep the language non-Turing-complete.
Really came here to search why EDN wasn't mentioned. It is used in Clojure/ ClojureScript/ hylang ... projects a lot. It is a superset of JSON, is in my opinion a lot more readable than JSON but familiar enough too. It has native sets e.g. #{1 2 "three" '("four element list with a string inside")} and keywords. Tagged elements can be used for extending e.g. with a timestamp (such as the built-in #inst) or #uuid. And it also supports comments and discards for stuff, that should be omitted in evaluation.
So one thing I wasnt sure of is: If you have a Starlark program how is the value of it decided? Is it simply the value of the last expression? And where does the print-output end up? Is it just for diagnostics and has no influence on the value?
IMO using code that generates (possibly binary/opaque) config data is the sweet spot. It's one more layer of indirection, but it means you're language-agnostic, you have a "safe" interface, and your "config-generating" process can be as expressive as you like -- comments, loops, whatever.
The underlying conundrum is:
- systems need to be configured,
- human-readability is obviously necessary at some level,
- configuration is often very "compressible" (needs loops, needs variables to be maintainable), but
- system-writers don't know the structure of your data, the axes on which you'd want to compress things, the best abstractions for you.
Templating languages are an obvious direction, but they're uniformly bad. If they have limited expressiveness you'll run into the limits. Maybe there are templating languages with good unit testing frameworks, but I haven't seen them. "Look at the expanded diff" doesn't scale. And generating gobs of human-readable "data" (in a format that supports comments!) is very wasteful.
I've tried dhall, cue and jsonet, and cue is so far my fav. It's very well designed, expressive, but restrictive enough so that config files don't look like scripts.
The way it blend types and values makes learning it super easy, yet you can do complex things with few lines.
But the main implementation exports to yaml without quoting the strings, which kinda defeat the purpose :(
I’m kind of unsure about the way CUE achieves reuse: if I understand correctly, you have files in a directory tree and the (result of processing the) bottommost files are the things you’re supposed to point your consuming tools at. So there’s no way to share structure among a collection of items if that collection is nested inside your config, the only operation available is essentially the generation of a set of similar but separate configs. Or am I wrong here? I’d very much like to be.
(Also, the type system is absolutely delicious, but it badly needs a paper with a complete description. I’m extremely interested in how it works, but fragmentary “notes on the formalism underlying” CUE are not enough.)
You do have imports and functions so you can reuse what you want.
The doc is also quite clear and rich, but the way it's organized means I have to read it entirely before writting my first CUE file. It also lacks IRL examples so trials and errors were my best friends.
CUE is based on Typed Feature Structures, which predate Deep NLP, and for which there is limited literature. We do need a good writeup on the theory. I've written a bit here: https://cuetorials.com/cueology/theory/
Think of a graph with lots of attributes within which paths are searched for.
I actually use CUE for large configuration files, used YAML before and had many issues once configurations became larger and more complex. It validates and exports JSON, which is easily readable in C++ and Python :-) have been happy doing the switch
Author seems to use misfeatures of a particular implementation to tar all implementations with. The round-tripping issue is not a statement about YAML as a markup language, much in the way a rendering bug in Firefox is not a statement about the web.
Stepping back a bit, YAML is good enough, and this problem has been incrementally bikeshedded since at least the 1970s, it is time to move on. Human-convenient interfaces (like YAML, bash, perl) are fundamentally messy because we are messy. They're prone to opinion and style, as if replacing some part or other will make the high level problem (that's us) go away. Fretting over perfection in UI is an utterly pointless waste of time.
I don't know what NestedText is and find it very difficulty to care, there are far more important problems in life to be concerned with than yet another incremental retake on serialization. I find it hard to consider contributions like this to be helpful or represent progress in any way.
If you can write a bad YAML document because of those mis-features/edge cases, I'd say you've already lost.
Humans are messy, but at the end of the day the data has to go to a program, so a concise and super simple interface has a lot of power to it for humans.
Working at a typical software company with average skill level engineers (including myself), no one likes writing YAML. But everyone is fine with JSON.
I think it's a case of conceptual purity vs what an average engineer would actually want to use. And JSON wins that. If YAML was really better than JSON, we'd all be using that right now.
So does it really matter if YAML is superior if >80% of engineers pick JSON instead?
I'd say that YAML is anything but conceptually pure, with all the arbitrariness, multitude of formattin options, and parsig magic happening without warning.
If you want conceptual purity (and far fewer footguns), take Dhall.
I would argue that you can write something poor and/or confusing in any markup language that is sufficiently powerful.
Conversely, if a markup language is strict enough to prevent every inconsistency, then it's not powerful enough or too cumbersome to use to be generally useful.
> Stepping back a bit, YAML is good enough, and this problem has been incrementally bikeshedded since at least the 1970s, it is time to move on
Nah, in the 1970s we had Lisp S-expressions that completely solved the problem, and everything since then has been regressions on S-expressions due to parenthesis phobia.
After hearing that thing about the country code for Norway, I became convinced that YAML has to just die. Become an ex-markup language. Pine for the fjords. Be a syntax that wouldn't VOOM if you put 4 million volts through it. Join the choir invisible, etc.
S-expressions don’t completely solve the problem: they don’t have a syntax for maps, and in practice there are at least two common incompatible conventions: alist or plist?
Solutions for this problem that I've used in my own S-expression config files:
1. Use only alists for maps because they prevent off-by-one errors.
2. Allow plists because they're less verbose than alists and use reader macros to distinguish them, and allow the reader macro definitions to be in the same file.
Most of the time I use option 1 because it's simpler.
I would argue that, in a data markup language, there shouldn't be a syntax for maps. Whether a given sequence should be treated as key-value pairs, and whether keys in that sequence are ordered or unordered, is something that is best defined by the schema, just like all other value types.
Obviously the application has to interpret the Lisp object resulting from reading the S-expression, just like it has to interpret any JSON, YAML, or anything else that it reads. So for maps you can, as you mention, use alists or plists. Regarding other stuff mentioned: none of the encodings are supposed to be bijective (the writer emits the exact input that the reader ingested). Otherwise, for example, they couldn't have comments, unless those ended up in the data somehow. There is ASN.1 DER if you want that, but ASN.1 is generally disastrous.
Stuff like escape chars were well specified in Lisps of the 1970s (at least the late 1970s), including in Scheme (1975). Floating point conversion is a different matter (it was even messier in the pre-IEEE 754 era than now) but I think the alternatives don't handle it well either. You probably have to use hexadecimal representation for binary floats. Maybe decimal floats will become more widely supported on future hardware.
A type-checked approach can be seen in XMonad, whose config files use Haskell's Read typeclass for the equivalent of typed S-expressions.
S-expressions don't solve the problem at all, you just get to fractally bikeshed all over again about what semantics they have and what transformations are or aren't equivalent. Does whitespace roundtrip through S-expressions? Who knows. Are numbers in S-expressions rounded to double precision on read/write? Umm, maybe. How do I escape a ) in one of my values? Hoo boy, pick any escape character you like and there's an implementation that does it.
>Human-convenient interfaces (like YAML, bash, perl) are fundamentally messy because we are messy
I don't know what to make of this statment, it has so much handwaving built-in. The most charitable interpretation I can find is that by 'Human-convenient' you simply meant the quick-and-dirty ideology expressed in Worse Is Better: Does job, makes users contemplate suicide only once per month, isn't too boat-rocking for current infrastructure and tooling.
Taken at face value (without special charitable parsing), this statement is trivially false. Python is often used as a paragon of 'Human-convenience', I sometimes find this trope tiring but whatever Python's merits and vices its _definitely_ NOT messy in design.
Perl is the C++ of scripting languages, it's a very [badly|un] designed language widely mocked by both language designers and users. Lua and tcl instead are languages literally created for the sole exact purpose of (non-) programmers expressing configuration inside of a fixed kernel of code created by other programmers, and look at their design : the whole of tcl's syntax and semantics is a single human-readable sentence, while lua thought it would be funny if 70% of the language involved dictionaries for some reason. These are extremely elegant and minimal designs, and they are brutally efficient and successful at their niches : tcl is EDA's and Network Administration's darling, and lua is used by game artists utterly uninterested in programming to express level design.
'Humans are messy' isn't a satisfactory way to put it. 'Humans love simple rules that get the job done' is more like it. But because the world is very complex and exception-laden, though, simple rules don't hug its contours well. There are two responses to this:
- you can declare it a free-for-all and just have people make up simple rules on the fly as situations come up, that's the Worse Is Better approach. It doesn't work for long because very soon the sheer mountain of simple rules interact and create lovecraftian horrors more complex than anything the world would have thrown at you. Remember that the world itself is animated by extremely simple rules (Maxwell's equations, Evolution by Natural Selection, etc...), it's the multitude and interaction of those simple rules that give it its gargantuan complexity and variety.
- you stop and think about The One Simple Rule To Rule All Rules, a kernel of order that can be extended and added to gradually, consistently and beautifully.
The first approach can be called the 'raster ideology', it's a way of approximating reality by dividing it into a huge part of small, simple 'pixels' and describing each one seperately by simple rules. I'm not sure it's 'easy' or 'convenient', maybe seductive. It promises you can always come up with more rules to describe new patterns and situations, and never ever throw away the old rules. This doesn't work if your problem is the sheer multitude and inconsistency of rules. The second approach is the 'vector ideology', it promises you that there is a small basis of simple rules that will describe your pattern in entirety, and can always be tweaked or added to (consistently!) when new patterns arise, the only catch is that you have to think hard about it first.
>and lua is used by game artists utterly uninterested in programming to express level design
Rather short sighted and dismissive to a successful programming language that's evolved over 20+ years. Lua is a great general purpose programming language that specializes not in "game making for non-programmers" but in ease of embedding, extension/extensability, and data description (like a config language). There's a whole section in Programming in Lua[1] to that effect. The fact that it's frequently used in games is credit to it's speed, size and great C API for embedding, not because of any particular catering to game designers.
You misunderstood me. I love lua and I wasn't being dismissive of it, I was using the first example that came to my mind to counter the claim that a convenient language has to be messy. Just because that was the example used doens't mean there is an implicit "and that's the only thing it's good for" clause I'm implying there: if someone said "Python is used by scientists utterly uninterested in programming to express numerical algorithms" would you understand that to be a dismissive remark against Python ?
Being used by non-programmers utterly uninterested in programming to solve problems is the highest honor any programming language can ever attain, because it means that the language is well-suited to the domain enough (or flexible enough to be made so) that describing problems in it is no different than writing thoughts or design documents in natural language. This is the single most flattering thing you can ever say about a language, not a dismissive remark.
Yes but I can understand the rationale. There are many numeric types and settling on some excludes use with others. If letting the application handle that, the configuration language can remain simple. His example where a version number 1.10 was round trip converted to 1.1 was enlightening.
The issue with YAML is that it does not unambiguously distinguish between number/booleans and strings. JSON does, but only for numbers, booleans, and nulls. But there are many data types that need to be conveyed. For example, dates and quantities (numbers with units, such as $3.14 or 47kΩ). Such things are left to the application to interpret. Even JSON does not unambiguously distinguish between integers and reals. Even so, JSON pays for its lack of ambiguity by requiring all strings to be quoted, which adds clutter and the requirement for quoting and escaping. Thus, supporting those extra types comes at a cost.
I think NestedText is unique in leaving all leaf values as strings, so it does not need quoting or escaping.
Everything involves a compromise. YAML provides a lack of clutter at the cost of ambiguity. JSON is unambiguous, but comes with visual clutter. In both cases there are still lots of types they cannot handle and so must be passed on to the application.
The compromise with NestedText is that it provides simplicity and a lack of clutter by not supporting any data types for leaf values other than string. Thus, all interpretation of the text is deferred to the application. But fundamentally that is the best place for it, because only the application understands the context and knows what is expected.
I have to say I hate the fact that I have low confidence when editing YAML that the result will be what I intend. It's kind of the number one job of such a format. And I routinely run into people using advanced features and then I have no idea at all how to safely edit it. It is interesting that it seems so difficult to pick a good tradeoff between flexibility and complexity with these kinds of languages.
I'm working on my own (you can start the attack hahaha). I feel there's not many simple, generic languages that allow to write simple DSLs with embedded documentation. The self documenting part is still missing but you can take a look and say what you think. It's kind of like yaml format (a bit TOML) with schema and possibility to merge multiple files with smaller chanks of the data. With export to json and yaml. https://github.com/dadlang/dadl
Have spent many years developing dev tools that use YAML and alternatives and I still think YAML wins because of its ubiquity relative interop with JSON. I’d pick HCL as an alternative if I was going to, as it’s been widely battle tested in Terraform.
JSON is often so hard to read I have to open a file in a formatting tool. It's also hard to tell if there's a simple missing quote or brace. Those are extra steps I don't have with YAML. For many things, JSON is great. For simple loaders, YAML gets it done quick and easy. They are functionally interoperable for the most part so I use what's best.
I'd also add the author has a competing framework so maybe there's a bit of pre-existing bias.
I'm not sure I'd trust manually editing either in a plain text editor. YAML cares about whitespace yet will silently make most combinations valid and it also has surprising pitfalls on type interpretation if one isn't consistently on top of it when entering data. JSON has a bunch of additional characters but at least if you forget them you have a higher chance of it just telling you it's wrong when you try to use it.
Thankfully it's very rare I ever have to open any config file in a text editor that isn't aware of dozens of formats so things like a missing brace are always suggested anyways and it isn't much a problem regardless of format.
When you miss a quote or a brace in JSON, the JSON fails to parse. When you make a similar minor mistake in YAML, you often end up with a valid but nonsensical document with completely incorrect structure.
I don't want the language to be flexible enough that simple common errors go unnoticed - I WANT the parser to tell me at parse time if I screwed something up. It's a similar dynamic to dynamic/static typing.
I don't understand this. YAML has limitations. All formats have limitations. If a format is too limiting, don't use it. Pick one more suitable, or come up with another one, like NestedText (or whatever). What is this need to tell everyone else to "move on" from using some format because it doesn't sit your specific preferences or use case?
I don't understand this. Opinions are not universal, every blogger has limitations. If you don't agree with a HN title, don't read it. Pick the next one or read another site. What is this need to tell everyone that you don't like them to tell everyone to "move on" just because it doesn't sit your specific preferences in formulation?
I see what you did there, but it comes off as mocking and not so clever.
The difference between my post and that blog, and your reply for that matter, is that I'm not telling anyone not to read it. I'm inviting you to comment on why people feel the need to tell others to move on.
The person that creates the config file does not necessarily choose the config file format. In the example, github chose YAML and everyone using github actions must use it. YAML is error prone, as everyone that tests with Python is finding out as the add Python 3.10 to their regression tests. This is a plea to organizations like github to stop choosing YAML.
As not a Python programmer, I'm struggling to understand the issue.
Your (or blogger's) claim is that GitHub misparses the YAML actions config when it comes specifically to Python? Or that YAML is inherently inadequate to the task of representing Python's necessary actions?
But now we discover that 3.10 is parsed as a float, just like the other ones were. But the problem is that 3.10 becomes 3.1, the equivalent float value! With 3.9 we didn't notice this problem.
I'm not sure what the name for this problem is.
It's something like.. an unfaithful but unintentionally working representation. Until it doesn't work anymore. The solution in YAML is to quote the values so that they become strings as intended.
How is that different from other formats? Take JSON:
{"python_versions": [3.8, 3.9, 3.10]}
This is a problem in any config language that doesn't enforce types, and jf it does enforce types, you should've used quotes already (like you really should've been before 3.10 was added to the list).
Similar problems also exist in many config formats with scientific notation (2e345) or hexadecimal notation (0x12345) or octal notation (012345, no that's not a decimal number in many programming languages and config formats!).
What commonly supported alternative would you suggest for this use case?
YAML's lack of limitations is the source of much of the difficulty with the format. The numerous ways to represent basic data (arrays, strings etc) is a common source of error. YAML doesn't have enough limitations!
And you can't pick what config format a tool you need uses.
But, GitHub, specifically has very specific guidelines on what to write and how to format it, and will give pretty detailed error messages if it's badly formatted. Not to mention tooling to format and highlight. I'm not seeing how the proposed NestedText is inherently free from those same issues: the need for tooling, guidance, error messages. Is the claim that it's easier?
I think the important difference between NestedText and YAML is that NestedText does not try to convert the text to numbers or booleans. YAML converts on to True and 3.10 to 3.1, which in this case is undesired. NestedText keeps text as strings. The idea is that end application should be the one that determines if on is a string or a boolean and whether 3.10 is a string or a number.
Its all in the name. All leaf values are strings. It is literally nested text.
XML in theory is a great format for what it represents — a tree of heterogeneous typed simple key/value pairs.
The problem is almost no data that people want to actually represent has this form and every way people have tried to beat XML into representing other things (i.e. lists and dicts) is kludgy is awkward.
I wrote a program at a corporate job where all the configuration is in Excel files. Tables are just fed into a dictionary and columns on each worksheet are predefined to hold the keys. People loved it because they know how to use excel and “text” is scary. (This is all very strange because they are just entering text in Excel, but the familiarity goes a long way)
I _recently_ suffered through a meeting where we developers were told to use the _new_ testing framework some team at our corp created. It's written in Java (we use .NET exclusively in our branch), configured via Excel sheets and Java, and exports results also as Excel sheets.
Whoever thought this was a good idea in 2021 has to be braindead. But the CEO was pleased. Probably because they know Excel.
I have a client that uses a CMS of unknown origins. I just get stuff placed in an s3 bucket, and then attempt to parse what was provided. 100% of their YAML files are invalid by every single linter I have found/tried. Not one of them understands where the error is occuring to help debug. It litterally just says invalid. I'm at a total loss. My head doesn't think YAML. Does a string need quoting or not? trailing spaces at the end causes problems? my personal experience in YAML is limited, but it hasn't been pleasant.
The trouble is that people insist in writing trees using plain text editors. Trees should be written in tree editors. Then you can't get the delimiters wrong.
"NestedText was inspired by YAML, but eschews its complexity. "
400 comments
[ 4.6 ms ] story [ 300 ms ] thread[1]: https://github.com/KenKundert/nestedtext
Or, NODE <name> .. END .
But I find it incredibly annoying to estimate indentation when lines are wrapped in an editor (or webpage). Or, to a lesser extent but still throws me off, when multiple blocks end at the same line. Or when pasting blocks into another block, and having to double check to make sure the indentation was carried over correctly. I like editors that visually show indentation characters.
it makes a difference on how it is parsed, so it's more than a devs preference
Personally I mostly use indent to read code, so requiring that the indent matches the semantic nesting makes it much easier for me to understand.
I get the original was talking about client facing config files. I'd rather see INI style config files personally.
If you're writing ugly code, braces or spaces won't save you. Just don't write ugly code.* Write it like the next person to view your code is an axe murder that knows where you live, so don't make them mad. You can minify later.
*I'm ignoring Perl, as it's always ugly
If it was code, an indentation error would often not compile or show errors or fail tests.
Configuration in YAML is much worse: most of the time an indentation error goes undetected until an application starts misbehaving.
Significant whitespace is perfectly ok for code but a huge footgun for YAML
My benchmark for yaml/JSON alternatives is "how would I feel explaining it to a busy, sceptical client?"
If the intended audience is purely developers, then sure. JSON (with the addition of comments and trailing commas) is just fine.
White space has the additional advantage of agreeing with itself. Other demarcations can have issues where the indentation and the structure contradict each other.
Please stop making code easy for non-coders. They don’t want to read it. They never did. They just want this damn box to work.
Just put the file in the root folder, and keep it as simple as possible and you should be fine? I mean, if they manage to write yaml correctly and consistently C is no match?
I wish it could be displayed with braces though, I wonder if someone already has built that as an extension for editing / viewing yaml files.
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
With that said if they weren't complaining about white space they'd be complainit about missing semicolons, missing/extra commas, missig equals signs, missing closing )]} or whatever.
I can definitely think of a broad range of people where I'd be happy to recommend they use text files for config and data but I wouldn't be happy if those text files needed to follow the rules of JSON syntax.
I mean - to some extent I would rather not edit JSON. It's not a terribly ergonomic experience. If I had to design a format for my own use, it would be indentation based and probably look a little bit like YAML, Markdown or similar.
I'm not clear exactly what the "it" here refers to but as I mentioned in other comments it's fairly self-evident that indentation is easier to visually parse than braces. A simple thought experiment - would you find it easier to skim read code where the indentation was consistent with the bracing or where it was inconsistent? Your brain registers the indentation first and you only resort to counting braces if there's a reason to doubt the former.
Nobody advocates for "braces are better without indentation".
This surely implies that (practically) everyone agrees indentation is carrying most of the weight of visual indication of structure.
Of course "everyone" might be wrong - but that's a fairly tricky corner to defend.
^^^ THIS ^^^
My benchmark is this: can an autoformatter do its job every time without breaking something that's technically working right now but possibly formatted wrong?
Every data format that cannot comply with this contains in it a huge waste of time. Even as a python programmer, I extend the same rule to programming languages.
Truncate YAML and in most cases you still have valid YAML.
I use it to generate the config files, and pass the generated config to the application the same way you’d pass any other json/yaml/etc to your app. Neither ever talk to each other, and Dhall doesn’t exist within our runtime environment.
https://xkcd.com/927/
This YAML:
is equivalent to this YAML: which is equivalent to this YAML/JSON:Basically take JSON, make quotes optional for strings and make curly brackets optional if the object is indented properly, and boom you've got (something like) YAML.
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
Yaml and toml are fine if you keep it simple. JSON direly needs comments support (but of course wasn’t designed to be used as a human config file format so that’s kind of on us). And not just “Jsonc that sometimes might work in places.”
Beyond that, I think we generally have all the things we need and I don’t personally think we need yet another yaml. =)
I also started advocating in-language configuration files (Python for Python, but also Lua for Lua, etc) a number of years ago because it lets you do really useful things (like functionally generating values, importing shared subsets of data, storing executable references, and ensuring that two keys return the same values without manual copy/paste) all without needing to spec and use Yet Another Thing™ that does only a fraction of what the programming language you're already using already does.
Things that get compiled can't really use it without recompilation.
* JSON streaming is a bit of a mess. You can either do JSONL, or keep the entire document in memory at once. I usually end up going with JSONL.
* JSON itself doesn't permit trailing commas. I can measure the amount of time that I've wasted re-opening JSON files after accidentally adding a comma in days, not hours.
* JSON has weakly specified numbers. The specification itself defines the number type symbolically, as (essentially) `[0-9]+`. It's consequently possible (and common) for different parsers to behave differently on large numbers. YAML also, unfortunately, has this problem.
* Similarly: JSON doesn't clearly specify how parsers should behave in the presence of duplicate keys. More opportunity for confusion and bugs.
Arrays with hole are a JS-only feature
JSON is defined by the spec. The people who wrote the spec think otherwise[0].
[0]: https://www.json.org/json-en.html
> Arrays with hole are a JS-only feature.
There are other langauges that allow arrays with missing elements.
The document > {1:2 3:[4 5]}
can only be "commafied" to > {1:2, 3:[4, 5]}
> There are other langauges that allow arrays with missing elements.
But javascript is the only one that gave JSON the JS in its name
I believe that's only true if one were to load YAML via the "SAX"-style per-event stream, and not the "object materialization" that normal apps use (aka `yaml.load_all` or JAX-B objects) since in those more data-object centric views, where would one put the processing events for those markers?
I also originally expected `yaml.parse(...)` to eat them as it does for comments and extraneous whitespace, but no, it does in fact return dedicated stream events for them, so TIL
On the other hand, XML allows comments, yet I've never seen XML parsers incompatibilities.
Those are not in compliance with the relevant spec. We need to treat them as damage and confront them on the technical and social level.
Easiest way to demonstrate how wrong that is, is to throw in a comment in the example document ;)
The desire to use a single interchange format for all data is the problem. There are plenty of reasons to support comments and minor syntax issues that JSON itself dislikes for human consumable and interactive JSON. I'd think software JSON could be just that.
For machine-to-machine generated payloads JSON is good enough.
While true, parsing it is still a minefield because it's very underspecified: http://seriot.ch/projects/parsing_json.html
Also, the changelog is quite frightening!
> [In 3.0], `[ ]` now represents a list that contains an empty string, whereas previously it represented an empty list.
* keys containing leading spaces, newlines or colons can be represented with the multiline key form, where each line of the key starts with `: `.
* leading or trailing space is not complicated; the string values are just the rest of the line after the separator from the key, `: `. The values are not trimmed.
* a string value beginning with `[` just works in most places. This would not be confused with list values, as these would only start after a new line. Only in the compact inline list and inline dict form there are restricted characters for syntax.
It seems that their claim, no escaping required, holds. The slightly more verbose form of the language constructs may be required to represent special values though.
[1]: https://nestedtext.org/en/stable/basic_syntax.html
It’s a deterministic subset of Python. This means that if you have complex or repetitive configurations, you can use loops and functions to structure them. But it’s impossible to write an infinite loop or recursion.
It's a cool idea to do configuration in a subset of Python but now you have to go implement that subset in every language.
I will look into this further.
Starlark is indeed deterministic and guaranteed to terminate (the Go implementation has a flag that allows recursion, but it's off by default), but these are two orthogonal properties.
If it has first-class functions, how can you avoid infinite recursion? Like, what stops me from running the omega combinator in it? This is why Meson (a similar language) does not allow those kinds of shenanigans, to keep the language non-Turing-complete.
[1] https://github.com/edn-format/edn
[2] https://github.com/cognitect/transit-format
As a sysadmin, YAML seems nice until you have actually done anything more advanced with it. See Julien Pivotto's presentation about some of its pitfalls: https://www.slideshare.net/roidelapluie/yaml-magic?next_slid... Btw. Jsonnet doesn't seem too bad either: https://www.youtube.com/watch?v=LiQnSZ4SOnw and here some examples: https://jsonnet.org/ but in my book, EDN still wins.
Or if you really must, then simply interrupt processes that loop infinitely, and fix the bugs that caused it.
You know, like you already do when you have an infinite loop.
Infinite loops are not the end of the world, you know. Processes can be interrupted, and computers have reset buttons.
It's still resources to detect loops, it often involves introspection or privileged views; it's simply easier to prevent loops.
The underlying conundrum is:
- systems need to be configured,
- human-readability is obviously necessary at some level,
- configuration is often very "compressible" (needs loops, needs variables to be maintainable), but
- system-writers don't know the structure of your data, the axes on which you'd want to compress things, the best abstractions for you.
Templating languages are an obvious direction, but they're uniformly bad. If they have limited expressiveness you'll run into the limits. Maybe there are templating languages with good unit testing frameworks, but I haven't seen them. "Look at the expanded diff" doesn't scale. And generating gobs of human-readable "data" (in a format that supports comments!) is very wasteful.
https://dhall-lang.org/
https://cuelang.org/
The way it blend types and values makes learning it super easy, yet you can do complex things with few lines.
But the main implementation exports to yaml without quoting the strings, which kinda defeat the purpose :(
(Also, the type system is absolutely delicious, but it badly needs a paper with a complete description. I’m extremely interested in how it works, but fragmentary “notes on the formalism underlying” CUE are not enough.)
The doc is also quite clear and rich, but the way it's organized means I have to read it entirely before writting my first CUE file. It also lacks IRL examples so trials and errors were my best friends.
It can be discouraging.
Think of a graph with lots of attributes within which paths are searched for.
Stepping back a bit, YAML is good enough, and this problem has been incrementally bikeshedded since at least the 1970s, it is time to move on. Human-convenient interfaces (like YAML, bash, perl) are fundamentally messy because we are messy. They're prone to opinion and style, as if replacing some part or other will make the high level problem (that's us) go away. Fretting over perfection in UI is an utterly pointless waste of time.
I don't know what NestedText is and find it very difficulty to care, there are far more important problems in life to be concerned with than yet another incremental retake on serialization. I find it hard to consider contributions like this to be helpful or represent progress in any way.
There's no canonical YAML implementation, and YAML spec is enormous (doubly so if you need to work with stuff like non-quoted strings etc. )
The formal grammar counts as canonical and several implementations are derived from it: https://github.com/yaml/yaml-reference-parser
If you can write a bad YAML document because of those mis-features/edge cases, I'd say you've already lost.
Humans are messy, but at the end of the day the data has to go to a program, so a concise and super simple interface has a lot of power to it for humans.
Working at a typical software company with average skill level engineers (including myself), no one likes writing YAML. But everyone is fine with JSON.
I think it's a case of conceptual purity vs what an average engineer would actually want to use. And JSON wins that. If YAML was really better than JSON, we'd all be using that right now.
So does it really matter if YAML is superior if >80% of engineers pick JSON instead?
If you want conceptual purity (and far fewer footguns), take Dhall.
Conversely, if a markup language is strict enough to prevent every inconsistency, then it's not powerful enough or too cumbersome to use to be generally useful.
YAML ain't markup language.
Nah, in the 1970s we had Lisp S-expressions that completely solved the problem, and everything since then has been regressions on S-expressions due to parenthesis phobia.
After hearing that thing about the country code for Norway, I became convinced that YAML has to just die. Become an ex-markup language. Pine for the fjords. Be a syntax that wouldn't VOOM if you put 4 million volts through it. Join the choir invisible, etc.
This is good: https://noyaml.com/
Erik Naggum had a notoriously NSFW rant about XML (over the top even for him) that I better not link to here, but lots of it applies to YAML as well.
1. Use only alists for maps because they prevent off-by-one errors.
2. Allow plists because they're less verbose than alists and use reader macros to distinguish them, and allow the reader macro definitions to be in the same file.
Most of the time I use option 1 because it's simpler.
Stuff like escape chars were well specified in Lisps of the 1970s (at least the late 1970s), including in Scheme (1975). Floating point conversion is a different matter (it was even messier in the pre-IEEE 754 era than now) but I think the alternatives don't handle it well either. You probably have to use hexadecimal representation for binary floats. Maybe decimal floats will become more widely supported on future hardware.
A type-checked approach can be seen in XMonad, whose config files use Haskell's Read typeclass for the equivalent of typed S-expressions.
(Sarcasm, just in case.)
YAML is patently unsuitable for any use case where the resulting output may require hand editing.
I don't know what to make of this statment, it has so much handwaving built-in. The most charitable interpretation I can find is that by 'Human-convenient' you simply meant the quick-and-dirty ideology expressed in Worse Is Better: Does job, makes users contemplate suicide only once per month, isn't too boat-rocking for current infrastructure and tooling.
Taken at face value (without special charitable parsing), this statement is trivially false. Python is often used as a paragon of 'Human-convenience', I sometimes find this trope tiring but whatever Python's merits and vices its _definitely_ NOT messy in design.
Perl is the C++ of scripting languages, it's a very [badly|un] designed language widely mocked by both language designers and users. Lua and tcl instead are languages literally created for the sole exact purpose of (non-) programmers expressing configuration inside of a fixed kernel of code created by other programmers, and look at their design : the whole of tcl's syntax and semantics is a single human-readable sentence, while lua thought it would be funny if 70% of the language involved dictionaries for some reason. These are extremely elegant and minimal designs, and they are brutally efficient and successful at their niches : tcl is EDA's and Network Administration's darling, and lua is used by game artists utterly uninterested in programming to express level design.
'Humans are messy' isn't a satisfactory way to put it. 'Humans love simple rules that get the job done' is more like it. But because the world is very complex and exception-laden, though, simple rules don't hug its contours well. There are two responses to this:
- you can declare it a free-for-all and just have people make up simple rules on the fly as situations come up, that's the Worse Is Better approach. It doesn't work for long because very soon the sheer mountain of simple rules interact and create lovecraftian horrors more complex than anything the world would have thrown at you. Remember that the world itself is animated by extremely simple rules (Maxwell's equations, Evolution by Natural Selection, etc...), it's the multitude and interaction of those simple rules that give it its gargantuan complexity and variety.
- you stop and think about The One Simple Rule To Rule All Rules, a kernel of order that can be extended and added to gradually, consistently and beautifully.
The first approach can be called the 'raster ideology', it's a way of approximating reality by dividing it into a huge part of small, simple 'pixels' and describing each one seperately by simple rules. I'm not sure it's 'easy' or 'convenient', maybe seductive. It promises you can always come up with more rules to describe new patterns and situations, and never ever throw away the old rules. This doesn't work if your problem is the sheer multitude and inconsistency of rules. The second approach is the 'vector ideology', it promises you that there is a small basis of simple rules that will describe your pattern in entirety, and can always be tweaked or added to (consistently!) when new patterns arise, the only catch is that you have to think hard about it first.
Rather short sighted and dismissive to a successful programming language that's evolved over 20+ years. Lua is a great general purpose programming language that specializes not in "game making for non-programmers" but in ease of embedding, extension/extensability, and data description (like a config language). There's a whole section in Programming in Lua[1] to that effect. The fact that it's frequently used in games is credit to it's speed, size and great C API for embedding, not because of any particular catering to game designers.
[1]: https://www.lua.org/pil/10.1.html
Being used by non-programmers utterly uninterested in programming to solve problems is the highest honor any programming language can ever attain, because it means that the language is well-suited to the domain enough (or flexible enough to be made so) that describing problems in it is no different than writing thoughts or design documents in natural language. This is the single most flattering thing you can ever say about a language, not a dismissive remark.
No numbers? Looks nice otherwise, but this seems like a very weird decision.
That said, JSON with its "I might overflow your number" attitude is not much better in this regard.
I think NestedText is unique in leaving all leaf values as strings, so it does not need quoting or escaping.
Everything involves a compromise. YAML provides a lack of clutter at the cost of ambiguity. JSON is unambiguous, but comes with visual clutter. In both cases there are still lots of types they cannot handle and so must be passed on to the application.
The compromise with NestedText is that it provides simplicity and a lack of clutter by not supporting any data types for leaf values other than string. Thus, all interpretation of the text is deferred to the application. But fundamentally that is the best place for it, because only the application understands the context and knows what is expected.
I'd also add the author has a competing framework so maybe there's a bit of pre-existing bias.
Thankfully it's very rare I ever have to open any config file in a text editor that isn't aware of dozens of formats so things like a missing brace are always suggested anyways and it isn't much a problem regardless of format.
I don't want the language to be flexible enough that simple common errors go unnoticed - I WANT the parser to tell me at parse time if I screwed something up. It's a similar dynamic to dynamic/static typing.
The difference between my post and that blog, and your reply for that matter, is that I'm not telling anyone not to read it. I'm inviting you to comment on why people feel the need to tell others to move on.
Here's a good example of how to respectfully disagree: https://news.ycombinator.com/item?id=29220994
Your (or blogger's) claim is that GitHub misparses the YAML actions config when it comes specifically to Python? Or that YAML is inherently inadequate to the task of representing Python's necessary actions?
I'm not sure what the name for this problem is.
It's something like.. an unfaithful but unintentionally working representation. Until it doesn't work anymore. The solution in YAML is to quote the values so that they become strings as intended.
Not reading the manual? Same thing would happen in JSON.
Sure it's annoying but all the YAML docs I've ever read state that unquoted numbers are ints/floats.
Similar problems also exist in many config formats with scientific notation (2e345) or hexadecimal notation (0x12345) or octal notation (012345, no that's not a decimal number in many programming languages and config formats!).
What commonly supported alternative would you suggest for this use case?
In YAML you can just write blah and and it becomes a string. Except when it doesn't, when it matches some other type of value.
Which means anything that accepts YAML accepts JSON.
So you can write your Github action code in JSON (with a yml file extension).
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
It sounds like the differences are unlikely to be encountered in conventional config files, especially those written by hand.
From that link I understand the source of problems to be:
Object keys longer than 1024 "stream characters". But these wouldn't be valid in any other yaml syntax either.
Some Unicode characters in strings (characters outside of BMP).
An escape sequence that is technically valid in JSON but never required.
And you can't pick what config format a tool you need uses.
Its all in the name. All leaf values are strings. It is literally nested text.
The problem is almost no data that people want to actually represent has this form and every way people have tried to beat XML into representing other things (i.e. lists and dicts) is kludgy is awkward.
Whoever thought this was a good idea in 2021 has to be braindead. But the CEO was pleased. Probably because they know Excel.
https://robotframework.org
Back in 2006, the testing was written in HTML tables, no idea how it manage to still be around.
"NestedText was inspired by YAML, but eschews its complexity. "
Mandatory XKCD: https://xkcd.com/927/
Fun fact: HTML was invented because SGML was "too complicated".
http://treeeditor.com/
https://github.com/josdejong/jsoneditor/