These are all valid points. But I still find YAML to be the best format for storing my strings for localizations. I find it much easier on my eyes than JSON. I’m open to other suggestions though.
I don't like NO as a boolean literal (mostly because it's also an ISO country code), but there's only one right way to parse it in a YAML 1.1 document. It was a mistake to apply the 1.2 rules to 1.1 documents and issue a "warning" about incorrect output, rather than require rejecting the document if the 1.1 rules aren't implemented.
To be fair, the article begins by linking to a previous article by the same author about the problems with JSON. He is an equal opportunities opponent :)
Aren't we just reinventing the wheel, though? Got your structured data format, now you need parsers (tons available for XML, incl SAX, DOM parsers, SimpleXML, Nokogiri...) a schema and validation tools (XSD), a templating mechanism (XSLT), a query language (XPath), ...
JSON was a reaction to the verbosity of XML, but a better reaction would have been to work harder on our text editors so that working with XML would be just as easy as working with JSON in terms of the numbers of keystrokes needed. Better parser interfaces that help you treat the dataformat more like it's part of the language would also help (i.e. SAX and DOMDocument suck to work with, but SimpleXML is almost idiomatic).
> JSON was a reaction to the verbosity of XML, but a better reaction would have been to work harder on our text editors so that working with XML would be just as easy as working with JSON in terms of the numbers of keystrokes needed.
Isn't that only solving half the problem? XML is also pretty difficult to read
But why is XML so freaking great? We can’t even tell if whitespace is significant or not. If a schema says it’s insignificant then that’s that!
https://www.oracle.com/technetwork/articles/wang-whitespace-...
That alone is TERRIBLE! (Same problem with YML.) Why should I bother with that? JSON can encode strings, hashes, arrays etc. in a way that’s instantly interoperable with JS and is far far more unambiguous.
What exactly is so great about XML that you can’t do with JSON in a better way? Schemas can be stored in JSON. XPATH can specified for JSON. Seriously I never got the appeal of XML except that it was first.
Two specific things about JSON: elements with the same name overwrite each other; and even though parsers are generally good about it, items are not required to be returned in the order they appear in the file.
One thing that bugs me about JSON is that it can't easily [0] represent general graph-structured data because its notion of identity is too limited. XML can represent general graphs trivially.
Some of XML's biggest achievements lie in written documentation formats(DocBook, DITA) where fine-grained markup control is needed and the presentation of the content is secondary to semantic features like footnotes, indexing, etc. These are formats that professional technical writers turn to when Markdown, Word docs or PDF won't quite do the trick.
For a lot of data, XML isn't the right form and buries too much data in hierarchy and tag soups - but it's flexible enough to make it into whatever you want, and since XML was buzzworded and XML libs were some of the easiest things to reach for in the 90's, it got pushed into every role imaginable.
If I have data that I need to send somewhere, and I can create the format for it, that's really easy to do.
The problem, every time, is the reverse; receiving some piece of data and trying to figure out what parts of it I care about. Both XML and JSON allow for schema definitions, but in both cases it fundamentally requires me, as a consumer "grokking" what is being sent. And the verbosity of XML simply makes that harder. Working with either is not _that_ hard (though I have run into XML in the wild that is so large a payload, yet so poorly designed, that there is no good way to process it; I can stream with via SAX without writing my own state handling mechanism, and I can't just deserialize it into an object without massive memory issues at scale); the difficulty really is in containing it in my mind, and JSON simply facilitates that better due to it's simplicity and explicitness (yes, explicitness; in XML it's not clear if a child element should only exist once, or multiple. JSON it's obvious)
Per the OP; I cringe every time I see YAML. Pain to write, pain to read; have to have tooling every time or I get whitespace issues.
I’d say this is schema-dependent. If you’re talking about plist files, sure; those are ugly and unintuitive. But on the whole I find XML far easier to read than JSON. With closing tags, what you lose in terseness is made up for with scannability: it’s easier to understand the document hierarchy at a glance, and find your place again after editing. Whereas with JSON I often have to match curly braces in my head, or add comments like `// /foo` which isn’t even possible outside of JS proper or a lax-parser environment.
I think that XML is often used for stuff that it shouldn't be used for. It could be almost OK (there are still a few problems though) for stuff containing text with other stuff inside that may in turn contain text, and so on. For other stuff, JSON or RDF or TSV or other formats can be good.
Verbosity certainly is an issue with XML, but far from the only one. IMO the main problem of XML is that it was designed as a markup language, but then misused as a data structure serialization language. When used as a markup language, the distinction between attributes and children is meaningful. When serializing data structures, the dichotomy breaks down. For most subfields of a larger data structure, it's not obvious whether to serialize that subfield as an attribute or as a child. Contrast with JSON, which only consists of obvious data structures.
> not obvious whether to serialize that subfield as an attribute or as a child.
Attributes are just strings, generally for metadata. I'd probably serialize an object from another language more verbosely. This is where an important distinction needs to be made: the XML format you use for config files or for data exchange from your app to others should not necessarily be just a serialized object from the most convenient form inside your application. If you care about the operators of the app, you'll allow them a more concise format for that kind of thing, and use XSD or your own internal mechanisms to turn that into an object you want to actually work with.
It's the sort of problem people have once, abstract away, and move on from.
It's also the sort of problem that just doesn't exist at all if you use a real data serialization format from the get-go.
I'm so frustrated with our collective attitude of building abstractions upon abstractions upon abstractions, without ever stepping back and realizing that we're using the wrong tool to begin with.
As far as I can make it, JSONs popularity grew from it being JavaScript which is, as we know, the knees bees. There was no big thinking in behind the whole thing and the role it plays today was certainly not the intended role (otherwise I cannot explain the non-standard data-formatting that is handled differently everywhere). JSON is more akin to Java RMI than to XML imho.
JSON was a reaction to the simplicity of having a data format that JS, ubiquitous on the web, could load via eval (which, once JSON was established, was largely abandoned because it is ludicrously unsafe, but the momentum was already there.)
SGML also has tag omission to make this even less verbose if desired. Or short references, which basically let you define arbitrary tokens SGML recognizes and replaces into something else, depending on the element context. These techniques in combination can be used to parse s-expr, CSV, markdown, and even some JSON, for example. Though personally I agree with others here that SGML is first and foremost a markup rather than config language.
The problem with tag omissions is that it requires that the parser have tag-specific information to properly build the AST. You can still do generic parsing with balanced </>.
I'd prefer XML because of the stability of the tools available for it.
Recently I was writing a custom static site generator for my website. I started with python and yaml using the pyyaml lib. After two months (don't laugh, I wasn't writing this generator all this time; i had a break) I tested if everything I wrote previously was warking. Pyyaml came at me screaming that they deprecated something and I shouldn't use it, otherwise the feds will get me.
Let's go several months earlier still. I was learning Python, using Debian Stretch which has Python 3.5 installed. The book I was learning from used Python 3.7. When I got to a point it became clear that 3.5 lacks a few things which are needed to continue learning Python according to the book. So I compiled Python 3.7, set up virtualenv with it and... The code I had previously written stopped working. With a version bump from 3.5 -> 3.7? That's a minor version change. And now 3.5 was expecting at one point to have a string path passed as an argument, and 3.7 was expecting a pathlib path. That was trivial to fix in case of a small example, but I would dread using such a thing for anything big and then having to debug what exactly broke between different (minor!) versions of Python or its libraries.
These new hip tools seem to have a backwards-compatibility issue.
I eventually settled on using XSLT and a couple really short shell scripts (which all fit on my screen at the same time) and I don't expect them to break in the next two decades.
However, XML is still a pain and I would prefer just using S-expressions and Lisp[1]. It's just that for now my only experience with them is writing things for Emacs and I would like to learn Scheme/CommonLisp to do anything outside of Emacs with Lisp.
This sounds more like a Python and pyyaml problem than a YAML or JSON problem. Someone could come along and totally refactor XSLT even if XML stayed the same, and then you would be in the same boat.
If you have libxml2-utils installed (package name may vary depending on your distro, but it almost certainly has a package) you can probably do something like "xmllint --format". You can also use "xmllint --shell file.xml" to get an interactive shell, or execute an xpath query and return, eg. "xmllint --xpath //foo file.xml". Use "-" for the file to read from stdin, as you might expect.
> With a version bump from 3.5 -> 3.7? That's a minor version change.
Python doesn't do semantic versioning. (Just like most language runtimes) You can find deprecations and backwards incompatible changes in the release notes.
I have to wonder if these stories are written by people who simply hate to type, or aren't good at it. Beyond that I can only chalk it up to academic curiosity.
XML is as pleasant to look at or touch as a nettle rash, but it seems it can join ALGOL 60 among the ranks of technologies which were a great improvement on their successors.
Namespaces are horrible in Python because the Python XML libraries are deficient. They're generally fine in e.g. Java. (Unless you're trying to represent QName typed data, but that's very niche.)
I used Python as an example, but handling namespaces in most languages besides Java (and even arguably in Java depending on your point of view) is rather painful.
namespaces are absolutely a bear and always unpleasant to work with. The libraries to use xml are equally frustrating when you’re doing complicated things, unless you want to make a class for every single type of detail that this xml document wants - then it’s fine, but some of us don’t want to do that, or inherited a project that didn’t do that.
XPath is a struggle with namespaces, as well. It’s ... trying.
How is that relevant for config files though? If you implement an app and want to use a config file, you don't have to use namespaces. I agree that namespaces are no fun, so I don't use them for my config files.
Namespaces are great when used wisely: they allow for embedding unrelated XML fragments in your document and your application will not see them. Or versioning your XML in namespaces. XPath and namespaces are trivial but it depends on the library you use on how to setup the namespace resolving. My pet peeve about namespaces: people that use HTTP URLs for their declaration and expecting them to be resolved for some reason. The other one being people that see a namespace prefix and think that that is the actual namespace.
I disagree. Back in the day we used attributes for everything that was key value and inner tags for anything with structure. We also formatted for clarity:
Putting l10n/i18n data inline is generally not a pleasant experience, and this problem persists regardless of what serialization format you are using. Either have separate data files per-locale that are merged with the defaults or store the localized strings in a central location (ala your usual gettext setup).
Yes there is. Markup languages are for representing rich text. Anything that gets rendered to the user is content and anything that's metadata (data about how to render) but not rendered as such goes into attributes. If there is no concept of "rendering to the user" in your app/data, then markup isn't the right choice for representing your data. You're blaming markup languages for not being designed to represent arbitrary data structures when you should be blaming yourself for misusing and misunderstanding markup. Though by skimming through this thread, XML appears to still do much better than YAML. And it's true that in the 00's, XML was improperly used and advocated as universal data format.
Any time the semantics represented by the XML is non-trivial, the schema design is likely to be obtuse and/or broken, even when done by smart people. XML is just hard to get right, and hard to evolve gracefully.
When you add verbosity on top of that, working with XML over the long haul is an utter pain.
XMLs "predecessor" is full-blown SGML, and since XML is just a proper subset of SGML, XML is no improvement. The one thing it brought was DTD-less, canonical angle-bracket markup where documents can be parsed without grammar rules eg DTDs (this was also incorporated into SGML via the WebSGML adaptations, so that XML could remain a proper subset of SGML). But this also means XML is useless for parsing HTML, the most important markup language (XML was started to supersede HTML as XHTML, but that failed).
I hate yaml but I have yet to find a better option for deeply nested confog files. Toml is the closest thing I have seen. Toml support is also not that great.
Yaml despite its flaws works pretty well for Ansible playbooks and for storing localizations.
Err, yes? Is quoting your keys, delimitation members with a comma instead of whitespace, and putting brackets/braces around collections really that confusing that people struggle to edit it by hand or read it in a diff?
I think the syntax is actually what makes it more human readable, it's still 95% text/numbers just annotated with information that makes it clear what things actually are instead of hiding them behind confusing computer parsing rules nobody is going to think about while reading human-friendly text.
I don't care for occasional config file writing but for stuff like ansible where writing YAML is the main job, I just wish it had chosen s other format but I guess it's too late.
Another one: Parsing partial YAML files doesn’t detect an error with loading the complete file. We’ve had a production outage, because of large yaml files getting cutoff and not all settings getting loaded into our server. JSON or XML typically will not parse.
YAML files should have an explicit start point and end point, respectively `---` and `...`. yamllint will enforce this, although I have never used these for integrity checks.
I've used YAML as the format for a config file, and I certainly regret that choice. Trying to explain to someone that doesn't know YAML how to edit it without setting them up for failure is quite annoying. There are too many non-obvious ways to screw up, like forgetting the space after the colon or of course bad indentation.
I’m not keen on how so many tools and services opt for YAML by default, either. Both JSON and YAML are a nightmare to handle once you’ve got 3000 line files and several layers of nesting.
CI would be a lot nicer to use if it didn’t rely on a single YAML file to work. And if you want to switch, suddenly you had a build step to convert back to YAML.
Your editor makes a world of difference here. Since you shouldn't be writing brace-language code without indents anyways, the biggest issue remaining is mixing tabs and spaces. Gedit makes this a big pain with it's default config (it doesn't even auto-indent) but Atom and IDLE handle it well.
Code you write yourself is not usually the source of problems with significant whitespace; it's situations like posting code on websites and discussing it where code in a whitespace-significant language becomes next-to-useless when leading whitespace is stripped, whereas code in any other language will still survive and then easily be autoformatted without changing its meaning.
Can’t remember the last time this has actually happened to me. In what websites are people posting code without code block formatting support? Like, instant messengers?
I’ve heard this argument in a lot of contexts, and it has always struck me as saying, “if hitting yourself with this bat hurts, try wrapping this towel around it and maybe it will hurt less.”
If I pinky promise to indent my code anyway, then why does it matter whether I also have braces or not? In fact, braces allow me to press one button in my editor and get the indentation absolutely perfect, without affecting semantics.
Braces also allow for easily copying and pasting blocks of code because the braces delimit the semantics of the copied text. Because your code is already indented, with white space indentation you have to check that a) you pasted the first line at the right indentation level b) every subsequent line is also at the right level relative to the first line. No small feat.
The main headaches are due to people either wanting to copy and paste code from various sites, or wanting to write really deeply nested code.
If you're writing well structured, original code in Python, it's generally cleaner and easier than other languages because the syntax avoids ambiguities that other languages have.
The difference in my experience is that once you know what's wrong with your whitespaces in Python, you're out of the woods. The interpreter is your friend from that point onward. YAML parsers, on the other hand, give you these really strange errors that are pretty difficult to understand, and it doesn't end with whitespaces.
There are quite a few comments saying they don't like python even from 10+ year users.
Language becomes popular largely through library ecosystem and resources around it, not just how the language looks. I think Google embracing it had a good role in acquiring mind shares.
YAML is easier to read and write. That's the benefit. It's also always going to be smaller than anything JSON or XML. Maybe it's not as correct, maybe some people don't like it, I don't really mind it. I don't see it really going anywhere soon either considering Kubernetes and the lack of alternatives in widespread usage.
I've never had someone that needed extensive help understanding YAML and that's besides reviewing work for people just coming up to speed. Find me an IDE or editor that doesn't have YAML support. Also, YAML supports comments so if you have pitfalls people need to know about you can document them inline.
Your argument is people who don't know things might screw stuff up. Well Yeah! This applies to everything.
> When they build something better I'll be the first to jump ship.
Will you though?
With learning a tool (almost any tool), its value increases with more use until you find a local maximum. The amount of effort to switch to something with a higher maximum at that point will definitely be considered by most people as part of the cost of that competitor. It is very hard to write off years of your life on anything.
When was the last time you did this?
It could take a few months at a minimum, just to see if you can understand it. Then, perhaps years of building things another way just to get good at it. Would you really give a few years of your life in making configuration suck less forever?
If you're serious, I have a potential solution: We don't have configuration files (or indention) or parsing problems, and users aren't the slightest bit confused on how to configure our applications (although they are often surprised if they've ever had to use a configuration file!). The downside is it's going to require a lot of re-learning on your part, and there's little I can do to make it any easier for you.
Will I? Personally, I enjoy tinkering with new tech. If we look at trends in technology, it's an absolute guarantee.
The last time I wrote off technology? In 30 years I've been doing this, I would say constantly, especially with tooling. I wouldn't consider the lessons learned using defunct tools to be lost either.
I'm not complaining about serialization formats, so why would I dedicate time to making it better?
Solution? I'm a quick learner so I'm not overly concerned. In a perfect world, you could just point me to your documentation.
Smalltalk environments don't typically have "configuration files" because it's so much easier to just directly manipulate the configuration parameter. Interactive development- the kind impossible with anything but the most specialised of IDE-- simply makes "config files" obsolete. Take a look at the seaside configuration guide[1] to get an idea what this is like.
In k/q[2] we also don't typically use configuration files, but for a different reason: Every type can be serialised over network or onto the disk, and when it's written to the disk it's usually in a format we can mmap() and access transparently. This is how q is also a database -- the data types q supports also includes tables.
k/q also has a built-in event loop that's not dissimilar to what you get when you run nodejs with the debugger port open except it's fast, and it's the regular way k/q processes communicate with each other.
What typically happens is that a table is designed for configuration, and we just expose it to the UI. Production environments are usually locked down so the only UI is to edit existing configuration parameters (and those are permissioned accordingly). These UI are typically quite general for any k/q data type, so they're quite rich and easy for people to use.
Then parts of the application interested in configuration just query the appropriate configuration table - this is only about 1000x faster than you would expect connecting to a remote database, and in many ways it's similar to a python application just storing its config in an sqlite database, except SQLite doesn't let you have a table as a data type so you can't put a table into another table, and you don't have tooling around comments and advice like you do with k/q UIs.
There are other places if you look carefully: Environments people have used (even beyond thirty years ago) that didn't have configuration "files", often had interesting and useful solutions to storing configuration. People tended to build configuration into part of their application, and so the legacy of that has tended to be excellent tooling instead of novel file formats.
You're trying to tell me a database or passing config options to your program are better than configuration files? What you're describing is the entire reason and purpose of why configuration files exist. YAML is also not limited to configuration files. If we look at the example you provided it's pretty clear:
"From the point of view of a component, the configuration can be thought of as a Dictionary-like collection"
This is exactly what YAML and JSON provide too. Using a configuration file is a choice, so are parameters, and so are databases. I'm not really understanding what you're trying to get at?
No. I'm saying when your application language is also your database or your operating environment [or for some other reasons], you don't need a configuration format.
The reason configuration file [formats] exist is because many programs are configurable and programmers are too lazy (or are not specified to, take your pick) to build a configuration tool [that has all their needs]. Configuration files are inferior in every way to an integrated and well-thought-out configuration process except that they may be easier to build and use in less ideal environments.
JSON is a fine format for interchange, and even persistence (i.e. to store configuration) but as a "configuration file" that people are expected to edit in their own way it is lacking, and that's why there are things like YAML and TOML and a million other things.
YAML is so bad for human writing. Everytime I write ansible tasks, I get confused with indentation and how to do arrays etc. JSON and YAML is frankly a generation behind compared to TOML or JSON5.
I think the author’s conclusion is in line with my own thought: If JSON is the problem, YAML isn’t the solution.
I recall the first time I saw YAML and all I could think to myself was that I have to learn yet another syntax. I find it far less readable than JSON or XML and made me pine for the latter.
The problem is with parsers, how they are implemented or used. YAML actually has a way to specify type of the data, alternatively the application supposed to suggest desired type. What's this take is showing is what types are assumed when they are not specified.
My vote is yes. Most configuration doesn’t need anything more sophisticated than key-value pairs, perhaps with namespaces. INI can manage that and TOML is basically a better-specified INI.
I can't tell if I've spent too much time on HN or if I came to this conclusion on my own, but TOML is my language of choice for configuration now. It's flexible in the right ways and sectioning of config is so important.
There's no white Knight here, they all suck in some way. Personally I've had decent success with yaml as simple configuration, but I would never use it as an interchange format. If you know it's caveats and you're targeting one language so you can become familiar with the parser it's serviceable.
If possible, prefer what tools in your vicinity use. My team uses Kubernetes and Concourse extensively, which both use YAML, so I tend to stick with YAML since people are already familiar with it.
(More recently, I've come around to prefer plain environment variables for configuration, but that only works nicely when the amount of configuration is fairly limited, say 20 values instead of 1000 values.)
The main issue I had with TOML is how much more syntactically noisy it is. Equivalent files with 2-3 levels of nesting usually become at least 50% longer than equivalent YAML.
This is a different use case, I think. This example is defining content, not configuration. In this case the content is user stories. I agree for creating sequences of documents/content in this way, YAML often is nicer. But for configuration, TOML is designed to specify it in a simple and flat way, and that can be very helpful.
I have some projects where I'm frequently writing and midifying content that resembles the example here, and I use YAML there and plan to keep using YAML. For most other things, I'm just doing configuration, so I use TOML. No reason you need to stick to one or the other.
My cursory survey of config / serialisation formats concluded that nothing is close to being good.
It's overly verbose, and hard to understand XML, it's no comments son, horrors of yaml or some okay format that doesn't have parsers for the languages (plural) you are using on your project.
I use JSON in the end. I prefer to write TOML, then parse that into JSON. This seems to strike a nice balance between human/machine write/read. It's simple enough to reason TOML, even if it gets verbose. If I have to write YAML after 2 layers I usually write it as JSON and include the JSON in the 2nd level of key.
In the scale world, HOCON is very nice. It’s a format designed explicitly for config files, and has a lot of niceties (like you can append files together and they merge correctly, so you don’t have to end up with giant config files)
I agree with HOCON being nice based on personal usage but I haven't seen an in depth analysis of it. This is the canonical parser for JVM based languages — https://github.com/lightbend/config, are there many other implementations that are widely used?
I think it's horses for courses. JSON I guess is the best for interchange i.e machine to machine, but I never want to edit it by hand; XML is relatively easy to read but can be quite painful to edit raw, but it can be quite easy to develop a structures editor. I’d favour it for document persistence. YAML is fine for configuration files but I would be careful about how I apply it and would always provide it as a heavily documented templated config file. YAML when used correctly is by far the easiest to edit in the clear, with a plain text editor. With that said, I would try to get away with basic namespaces properties files first before I’d go that far ...
ini if needs are crazy simple, YAML if you need a structure like JSON's but with something any human ever needs to interact with. JSON if humans aren't in the loop.
TOML, in my opinion, is like a weird mishmash of JSON, ini, and bashisms. Though I have worked with it a lot less than the other formats, so YMMV.
Try writing strings with backslashes, or adding a new line to your array and accidentally leaving a trailing comma (or accidentally forgetting it for the previous row). It's also just very visually noisy. I agree with a lot of the other comments in this thread that for human configuration: TOML > YAML > JSON > XML
You can edit JSON by hand but it’s not what it’s for. It’s not designed for that and it’s not really suitable for that. Theoretically you can milk anything that has nipples, but you might find the experience of milking a cat to be ... challenging
.ini, followed by TOML, followed by an identical implementation of some other app's config format.
The biggest problem with config formats is they mislead users into thinking they understand the format. The user tries to edit it by hand, and chaos ensues. So only formats that are stupidly simple, or whose warts are already familiar and well documented, are good choices.
Apache had a great configuration format. Nothing else used it (that I knew of) but you could in theory implement "Apache configs" and then people'd just have to look up how to write those, which there's lots of examples of.
JSON and YAML and XML are data formats; they should only be written by machines, and read by humans. Same with protocols like HTTP, Telnet, FTP... You're not supposed to write it yourself, but it's readable to make troubleshooting easier.
Data formats are nice for expressing nested data structures, but then they don't (usually) support logical expressions; at that point you need a template/macro/programming language, and at that point you're writing code, which will need to be tested, and at that point you should just write modules and use a config format to give them arguments. Every complex tool goes through the same evolution.
If you care about your users, write a tool to generate configs based on a wizard. Good CLI tools do this, and it really makes life better. (It's also a great way to document all your config features in code, and test them)
I say just use JSON. Everyone knows it already and it's good enough. Use a parser in your app that allows comments and trailing commas like vscode does.
For in-house, python-only project, my way to go is to create a "config.py". Then I declare a bunch of module variables that can be overridden by environment variables as a bonus.
Disclosure: I work on Tree Notation. It’s the future of file formats, IMO.
The idea is to have 2 levels: a simple, minimal syntax/notation (think binary) called Tree Notation, and then have higher level grammars on top of that, called tree languages.
It works for encoding data and also for programming languages, regardless of paradigm.
Your project looks interesting, but I looked through the Github project and site, but I couldn't find a language specification, reference manual, BNF-like grammar, or anything to indicate what the syntax is, beyond a very trivial example in the Github. To be blunt, I think you need to start with a spec to get any traction. That allows people to understand your data model and particular text encoding of it. If they like it, they might use your tool and perhaps port the system to other languages.
Docs needs work, in particular I’m hoping people will create their own explanations of the ideas in external places, as that might be a better way to understand it. Happy to provide help to anyone that is interested in that.
Edit: we've had to ask you multiple times already not to be a jerk on HN. Would you please review the guidelines and take the spirit of this site more to heart?
From a quick glance, my issue with Tree Notation would be that it's not enough syntax, i.e. it does not provide enough structure for me to grasp the overall structure with a cursory glance. Maybe it would work better if GitHub had a syntax highlighting for it. (But requiring syntax highlighting to be readable is a large red flag on its own.) Or that's just a feeling that would be mitigated if I saw larger files.
GitHub syntax highlighting is coming. If anyone wants to help with that that would be awesome. One of the top outstanding issues. Syntax highlighting is there for sublime and codemirror but one bug left in those implementations and would love help getting syntax highlighting generation going for Monaco and linguist so we’d have it everywhere on GitHub.
Your impression is correct though, without highlighting it’s awful. Try the Tree Language designer app to see it with highlighting, type checking, autocomplete etc...there are interesting ways to accomplish everything without syntax, often not obvious. But tooling is essential to make it better than existing options. Help wanted!
I find yaml to be the best for making cloudformation templates. On its own it isn’t much good but if you use the right plugins it really is better than json.
Sort of, I deliver a GUI that exports into YAML for pretty much only reading, portability, and version control. People are expected to do the editing in the GUI, only using YAML for editing when doing complex regex operations that my GUI doesn't support.
If people aren't editing it by hand, why does the format matter? Why not just use JSON? Tools for too-complex-for-the-GUI manipulations are at least as good for JSON as they are for YAML, and the editing is less error-prone.
The superior replacement for XML, JSON and YAML is the SQLite .db file. Easy to “parse”, easy to manipulate programmatically, what more could you want?
This refrain just cheeses me right off every time. Nothing is human readable! Everything requires a program to read it, because no human being can read states of charge or states of magnetic polarization directly.
What makes something 'human readable' or not is a software tool. Underlying that tool is a data format that the tool can accept and display. What everyone means when they try to sound smart by saying 'human readable' is just 'plain text.' In other words, they know where to find the dumbest possible reader/editor for it. Text editors are the dumbest possible editors because they cannot constrain edits to conform with the grammar of the interface language; they allow bugs at a point in the development process where it is trivial to disallow bugs, especially considering that interface languages should probably be, at most, regular languages.
I've a certain sympathy for your position but I interpret "human readable" as meaning readable and somewhat comprehensible without using specialized tools.
But when is a tool specialized? Would a dedicated XML / JSON / YAML / ASN.1+DER / Avro / Protobuf / Parquet editor be specialized? And what if that hierarchical standard would be the de facto industry standard? Is a binary file editor specialized? What about a structured assembler?
Personally, I think most editors are rather specialized. They deduce the character sets, often add syntax highlighting and provides paging for very large files. High level strongly typed languages, such as modern C#, Java, Scala are designed to be used in an IDE. You could view and edit it, but it provides a difficult situation (not unlike editing XML by hand).
"Human readable" is very subjective. It depends on the person, the task at hand, the intended recipients, etc. etc.
Perhaps you should stand on that soapbox a little more often. People are way to enamored with their little 1970's ed and ex derivatives.
Every veteran in the field knows that data and data structure are the primary enablers for almost any solution. Nevertheless, we have regressed in the last decades w.r.t. that. Had we had better structured (AST-driven) editors and context dependent representation, XML (or sth similar) might have taken off.
Personally, I blame the exponential increase in inflow of new developers. Nothing but reinventing the wheel without knowing history.
As an ansible user, I hate YAML and its broken parsers with a passion, but the security objection does not make much sense. It does apply verbatim to any parser of anything if the implementation decides that a given label means "eval this content right away". I fail to see how this can be a fault of the DDL rather than the parser's.
The reason this is a fault of the DDL and not the parser is that the DDL spec decides that it has label that evaluates a command. The parser then has two options, either implement it or not conform to the spec (and essentially implementing a different DDL). For programming languages it makes sense to have an eval label/command. For configuration/serialization DDLs I think it's a terrible choice.
And terrible it is indeed, but I cannot find it specified - the strings eval, exec, command, statement do not even occur in the official specs (shallow doc perusal, I know)
> As an ansible user, I hate YAML and its broken parsers with a passion
Could you elaborate on this? I use Ansible daily and I've never had a problem with YAML once I took some time to understand it. What do you mean by broken parsers? I'm assuming that's something Ansible specific you are referring to.
I intensely dislike yaml's whitespace-based syntax because whitespace is white, and it gives very little visual context expecially in long, nested documents. Editors that expand/collapse branches do help some, but are no match for highlighting matching pair of braces in other saner formats/languages (I am also not a fan of syntactic whitespace in python, if you get my drift.)
And ansible's parser is broken, in more ways that I can remember (haven't been writing playbooks and stuff for a couple of months now). If you like pointless pain, try embedding ":" in task names for a demo (or one of other several "meta" characters: the colon is just the one that ends to recur most).
I will give a passing mention to the smug, vague error message "You have an error at position (somewhere in the middle of the file) It seems that you are missing... (something) we may be wrong (they almost always are) but it appears it begins in position (some position close to the first line)" that sets off a hunt for the missing brace/colon/space/whatever and makes me want to do stuff to the person who devised it.
This compounds with the confusion brought on weaving of yaml's and jinjia2 syntaxes and ansible's own flakiness on deciding what is evaluated when - which decides when and if a variable does indeed change, when does yes means "yes" rather than true or 1, but not '1' or "true" (try prompting the user for a boolean variable, and find yourself writing if ( (switch == "true") or (switch == "1")) in short order).
Pity that ansible is so damn convenient, or I would have ditched it long time ago for anything - bash included (OK, maybe not bash).
I strongly recommend doing away with config files completely for sake of ease of use, maintainability and security.
Instead just declare all config variables within code itself in a separate config class/module file, along with initialization to default values and provides dynamic getter/setter interface over a debug API (which can be enabled/disabled via a command line flag).
If you want, you can also provide a friendly cli tool to interact with the debug api. This tool could output help messages, show current config values - differentiate between default vs overwritten etc.
Of course, this can be written once as a utility library and cli and used consistently across all your programs.
> If you want, you can also provide a friendly cli tool to interact with the debug api. This tool could output help messages, show current config values - differentiate between default vs overwritten etc.
And we load the config by curling a JSON payload :) ?
For the love of god, if anyone reads this, please don't do this!
Config files are fantastic. Trivial to read, write, copy, track in version control, diff, grep, generate with scripts, etc.
API-driven configuration has none of these properties.
Some Java application servers take this approach of API-driven configuration. It's an improvement over UI-driven configuration, which is what they had before. But it's still significantly worse than simple file-driven configuration.
If you want to provide a 'friendly' CLI tool, by all means do so - but provide a tool to interpret and generate config files, not something which replaces config files.
> Config files are fantastic. Trivial to read, write, copy, track in version control, diff, grep, generate with scripts, etc.
Code based configuration offers most of those benefits, just look at some of the suckless tools, if you've never used dwm or written a line of C I bet you can still guess
how to configure some stuff here: https://git.suckless.org/dwm/file/config.def.h.html .
It doesn't work for all software of course, but for anything developer centric and/or anything with complex configuration done by experts it's a good choice, particularly for any software going down the path of creating a custom language (tmux, vim, etc). You get the power and flexibility of a full programming language, you get compile time config checks, you get excellent performance, you don't have to learn yet another config language and it's easy to write patches.
It's not great if you're targeting Joe Average, but neither is any configuration format, or any configuration at all.
Config variables in code gets version controlled and release-managed alongside code and takes the same CI/CD route to production as rest of your code does. Your code asserts, compilers and test cases can help you catch errors in config data. All of this happens without any special file format or parser concerns.
If you need runtime reconfiguration in production, then it requires a runtime config management system tailored for operations folks with proper authn/authz, audit logs etc. This is a product by itself.
The connection between your running program and the runtime configurator has to be intentional, secure etc.
IMO, config variables should have following bindings:
1. config variable in code.
2. program start env variable.
3. program start command argument flag.
4. runtime configuration.
All available configuration options are declared in the code. But not all configuration options should be accessible from #2 to #4. And the override preference order may not be same for all types of configuration variables.
Btw, this isn't related to Java or any particular programming language. I've seen this done in large C++ projects 20 years ago.
I went from that to INI to TOML and back to that in 10 years.
It just makes things run smooth in TypeScript because all the members and types can be statically analyzed giving you errors, auto completion and ability to jump to the definition when config files cannot do that but probably not good for large projects with multiple languages.
For small projects I'm just taking the benefit over small concerns.
I'll say it: I think YAML is great and a joy to use for configuration files. I can write it even with the dumbest editor, I can write comments, multi-line strings, I can get autocompletion and validation with JSON schema, I can share and reference other values. It allows tools to have config schemas that read like a natural domain specific language, but you already know the syntax. I haven't had problems with it at all.
This was me too - until yesterday, when I made a minor change to one of our YAML config files and everything broke. On investigation it turned out that all of our YAML files had longstanding errors but those errors happened to be valid syntax and also did not cause any bad side effects, so we had been getting away with it by pure luck until I made a change that happened to expose the problem.
That would make me not a fan of the particular parsers/validators I've been using, rather than not a fan of YAML.
The big strike against YAML I see there is that it needs a good conformance test suite and implementations need to be tested against it. But that's not a problem with the format but a fairly easy to fix ecosystem problem.
I've got to say it is the most frustrating config file ever to wrote. The only time I have to use it is for Docker Compose and I am constantly fighting vim on indentation and trying to make sense of confusing errors about "unexpected block start." Do you have any suggested vimrc for YAML?
I agree. As long as you're using a strict parser, I've found YAML to be much nicer for configuration than JSON. I use Python's ruamel.yaml library, and have never had any weird type problems. Once the nesting gets too deep, it can be a pain. but that's the same for JSON.
I have found myself using TOML more and more for configuration, though. It helps a lot with keeping things flat and easy to read. I'll still prefer YAML over JSON for human-writable files, but I'm starting to prefer TOML over YAML.
That's really close to [RFC 7464](https://tools.ietf.org/html/rfc7464), JSON Text Sequences. It uses U+001E RECORD SEPARATOR. The `jq` tool supports those if you pass a flag.
Having to close contexts is a VERY good 'sanity check' to see if something is malformed or not.
If appending is necessary make the parser handle multiple copies of the namespace and merge them upon output. Unknown keys and sections should also always be copied from input to output (this is how you embed comments).
Not sure about other languages and libraries, but Go supports this out of the box[1]. And while we're at it, why not CSV? That can be processed with awk.
I use JSON streams a lot with command line tools. Keep in mind that it’s limited in that each object must consume a single line. This allows you to recover from syntax errors in a single entry; each line is a fresh start.
Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.
Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config.
[{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}]
A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...)
Try just using line-delimited JSON objects (http://jsonlines.org/). It ticks all of your boxes, especially 3: "jq -s '.cmd' fish_history | histogram".
Neither YAML or Protobufs are quite as easy as that.
All in all it's ridiculously simple, easy to parse in a variety of languages and each row is a single line that's simple to iteratively parse without loading the whole thing into memory.
this seems like it makes json useful for logging, but not too useful as configuration. For instance, it doesn't support commenting, and it seems like every line needs to have all its children compressed onto one line?
Tcl with control structure commands disabled and infix assignment for convenience. Jim Tcl is a lightweight implementation if the main line isn't workable.
I haven't run into that. I think in the contexts for which TOML was meant, it isn't that big of an issue though. Also, I'm not entirely sure how many parsers enforce that restriction.
TOML would be great, if not for an annoying obscure detail in the specification that makes it hard to use for my typical use cases (scientific computation) [1]. Moreover, I find quite unintuitive how you are supposed to specify array of tables [2]: this kind of is much easier in JSON (which is the format I am currently using, although it is far from perfect).
That is an annoyingly obscure detail, I think the wrong decision was made there. Hopefully it gets reversed.
Personally I really like the array of tables syntax. It is a little unintuitive but it's not difficult to remember. It's useful for fulfilling the OP's "No document closing necessary, so appending is trivial" requirement.
And if you don't want to use it, you can always use inline tables in an array, just like JSON.
S-Expressions are quite simple, there are some parsers floating around in well-known projects, although I'm not sure they're SAX-style: https://leon.bottou.org/projects/minilisp
I also wonder if you need a text format, or if SQLite or systemd's journal API would work.
I love proto, but the textformat was an after thought. The binary format is rigorously defined, portable, extensible and optimized. The text format was reverse engineered from the c++ implementation after the fact when folks found textproto useful. Unfortunately there are discrepancies between languages around the corner cases of the textformat and that's the sad world we live in. Avoid letting textproto be part of your user exposed interface.
The security problem is common to many serialisation formats and similarly terrible bugs have happened in a large number of formats.
For instance, the recent iMessage bugs that project zero announced were because NSCodable serialization tells the deserializer what class should instantiated. Followed by remote code execution (woo!)
Similar problems have occurred with java serialization over the years, the python serialisation thing (that silly name I can’t recall).
I was recently learning swift and was getting frustrated by the verbosity/work for deserialisaing abstract classes when I realized the clunkiness was due to a design that made the deserialise attacker specified objects basically impossible. Obviously you could engineer a solution that would be exploitable but there’s only so much a platform can do to stop developer mistakes.
I learned some of the intricacies of yaml the other day when refactoring a docker-compose project. At first glance, it's brilliant... Until I started running into limitations, edge cases, and issues.
I like the _idea_ of yaml, but:
- it's overly complicated in the wrong ways
- common/simple use cases aren't supported and require post processing (i.e. Merging block maps/arrays, string interpolation, etc)
456 comments
[ 2.8 ms ] story [ 351 ms ] threadJSON was a reaction to the verbosity of XML, but a better reaction would have been to work harder on our text editors so that working with XML would be just as easy as working with JSON in terms of the numbers of keystrokes needed. Better parser interfaces that help you treat the dataformat more like it's part of the language would also help (i.e. SAX and DOMDocument suck to work with, but SimpleXML is almost idiomatic).
Isn't that only solving half the problem? XML is also pretty difficult to read
But why is XML so freaking great? We can’t even tell if whitespace is significant or not. If a schema says it’s insignificant then that’s that!
https://www.oracle.com/technetwork/articles/wang-whitespace-... That alone is TERRIBLE! (Same problem with YML.) Why should I bother with that? JSON can encode strings, hashes, arrays etc. in a way that’s instantly interoperable with JS and is far far more unambiguous.
What exactly is so great about XML that you can’t do with JSON in a better way? Schemas can be stored in JSON. XPATH can specified for JSON. Seriously I never got the appeal of XML except that it was first.
Oh and there's no comments.
[0] https://realprogrammer.wordpress.com/2012/08/17/json-graph-s...
For a lot of data, XML isn't the right form and buries too much data in hierarchy and tag soups - but it's flexible enough to make it into whatever you want, and since XML was buzzworded and XML libs were some of the easiest things to reach for in the 90's, it got pushed into every role imaginable.
If I have data that I need to send somewhere, and I can create the format for it, that's really easy to do.
The problem, every time, is the reverse; receiving some piece of data and trying to figure out what parts of it I care about. Both XML and JSON allow for schema definitions, but in both cases it fundamentally requires me, as a consumer "grokking" what is being sent. And the verbosity of XML simply makes that harder. Working with either is not _that_ hard (though I have run into XML in the wild that is so large a payload, yet so poorly designed, that there is no good way to process it; I can stream with via SAX without writing my own state handling mechanism, and I can't just deserialize it into an object without massive memory issues at scale); the difficulty really is in containing it in my mind, and JSON simply facilitates that better due to it's simplicity and explicitness (yes, explicitness; in XML it's not clear if a child element should only exist once, or multiple. JSON it's obvious)
Per the OP; I cringe every time I see YAML. Pain to write, pain to read; have to have tooling every time or I get whitespace issues.
https://json-schema.org/
I’d say this is schema-dependent. If you’re talking about plist files, sure; those are ugly and unintuitive. But on the whole I find XML far easier to read than JSON. With closing tags, what you lose in terseness is made up for with scannability: it’s easier to understand the document hierarchy at a glance, and find your place again after editing. Whereas with JSON I often have to match curly braces in my head, or add comments like `// /foo` which isn’t even possible outside of JS proper or a lax-parser environment.
Attributes are just strings, generally for metadata. I'd probably serialize an object from another language more verbosely. This is where an important distinction needs to be made: the XML format you use for config files or for data exchange from your app to others should not necessarily be just a serialized object from the most convenient form inside your application. If you care about the operators of the app, you'll allow them a more concise format for that kind of thing, and use XSD or your own internal mechanisms to turn that into an object you want to actually work with.
It's the sort of problem people have once, abstract away, and move on from.
I'm so frustrated with our collective attitude of building abstractions upon abstractions upon abstractions, without ever stepping back and realizing that we're using the wrong tool to begin with.
JSON was a reaction to the simplicity of having a data format that JS, ubiquitous on the web, could load via eval (which, once JSON was established, was largely abandoned because it is ludicrously unsafe, but the momentum was already there.)
(Mostly /s. Come at me:))
My one request would be to bring back to SGML-like closing tag abbreviation:
That is, instead of
we should be able to write I think this one change would make XML more "palatable" for the JSON/YAML/TOML crowd.Recently I was writing a custom static site generator for my website. I started with python and yaml using the pyyaml lib. After two months (don't laugh, I wasn't writing this generator all this time; i had a break) I tested if everything I wrote previously was warking. Pyyaml came at me screaming that they deprecated something and I shouldn't use it, otherwise the feds will get me.
Let's go several months earlier still. I was learning Python, using Debian Stretch which has Python 3.5 installed. The book I was learning from used Python 3.7. When I got to a point it became clear that 3.5 lacks a few things which are needed to continue learning Python according to the book. So I compiled Python 3.7, set up virtualenv with it and... The code I had previously written stopped working. With a version bump from 3.5 -> 3.7? That's a minor version change. And now 3.5 was expecting at one point to have a string path passed as an argument, and 3.7 was expecting a pathlib path. That was trivial to fix in case of a small example, but I would dread using such a thing for anything big and then having to debug what exactly broke between different (minor!) versions of Python or its libraries.
These new hip tools seem to have a backwards-compatibility issue.
I eventually settled on using XSLT and a couple really short shell scripts (which all fit on my screen at the same time) and I don't expect them to break in the next two decades.
However, XML is still a pain and I would prefer just using S-expressions and Lisp[1]. It's just that for now my only experience with them is writing things for Emacs and I would like to learn Scheme/CommonLisp to do anything outside of Emacs with Lisp.
[1] https://sites.google.com/site/steveyegge2/the-emacs-problem
> I'd prefer XML because of the stability of the tools available for it.
Python doesn't do semantic versioning. (Just like most language runtimes) You can find deprecations and backwards incompatible changes in the release notes.
Although those breaking changes are not frequent and if https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep... resulted in broken code, you likely ran into something that's reportable as a bug.
XPath is a struggle with namespaces, as well. It’s ... trying.
I disagree. Back in the day we used attributes for everything that was key value and inner tags for anything with structure. We also formatted for clarity:
Compared with what we used to do, I look at attribute-less maven pom.xml with horror.[0]https://www.w3.org/TR/REC-xml/#sec-lang-tag
It's your job to decide how the data should be structured, in any language.
I would do the following :
When you add verbosity on top of that, working with XML over the long haul is an utter pain.
Yaml despite its flaws works pretty well for Ansible playbooks and for storing localizations.
I think the syntax is actually what makes it more human readable, it's still 95% text/numbers just annotated with information that makes it clear what things actually are instead of hiding them behind confusing computer parsing rules nobody is going to think about while reading human-friendly text.
https://yamllint.readthedocs.io/en/stable/rules.html#module-...
CI would be a lot nicer to use if it didn’t rely on a single YAML file to work. And if you want to switch, suddenly you had a build step to convert back to YAML.
Braces also allow for easily copying and pasting blocks of code because the braces delimit the semantics of the copied text. Because your code is already indented, with white space indentation you have to check that a) you pasted the first line at the right indentation level b) every subsequent line is also at the right level relative to the first line. No small feat.
Without braces copy-pasting becomes a context-sensitive headache. That's poor usability.
Quite the opposite.
I like to format my code nicely anyways (or rather, mostly my editor does it for me because I’ve asked it to do so).
I indent with two spaces usually, regardless of language. And have my editors configured to insert two spaces when I press tab.
JavaScript, Rust, Python, C. Same difference, in terms of how I use whitespace.
If you're writing well structured, original code in Python, it's generally cleaner and easier than other languages because the syntax avoids ambiguities that other languages have.
Language becomes popular largely through library ecosystem and resources around it, not just how the language looks. I think Google embracing it had a good role in acquiring mind shares.
https://news.ycombinator.com/item?id=20672051
I've never had someone that needed extensive help understanding YAML and that's besides reviewing work for people just coming up to speed. Find me an IDE or editor that doesn't have YAML support. Also, YAML supports comments so if you have pitfalls people need to know about you can document them inline.
Your argument is people who don't know things might screw stuff up. Well Yeah! This applies to everything.
You may be surprised to find that there’s significant disagreement on that point.
Will you though?
With learning a tool (almost any tool), its value increases with more use until you find a local maximum. The amount of effort to switch to something with a higher maximum at that point will definitely be considered by most people as part of the cost of that competitor. It is very hard to write off years of your life on anything.
When was the last time you did this?
It could take a few months at a minimum, just to see if you can understand it. Then, perhaps years of building things another way just to get good at it. Would you really give a few years of your life in making configuration suck less forever?
If you're serious, I have a potential solution: We don't have configuration files (or indention) or parsing problems, and users aren't the slightest bit confused on how to configure our applications (although they are often surprised if they've ever had to use a configuration file!). The downside is it's going to require a lot of re-learning on your part, and there's little I can do to make it any easier for you.
The last time I wrote off technology? In 30 years I've been doing this, I would say constantly, especially with tooling. I wouldn't consider the lessons learned using defunct tools to be lost either.
I'm not complaining about serialization formats, so why would I dedicate time to making it better?
Solution? I'm a quick learner so I'm not overly concerned. In a perfect world, you could just point me to your documentation.
Smalltalk environments don't typically have "configuration files" because it's so much easier to just directly manipulate the configuration parameter. Interactive development- the kind impossible with anything but the most specialised of IDE-- simply makes "config files" obsolete. Take a look at the seaside configuration guide[1] to get an idea what this is like.
In k/q[2] we also don't typically use configuration files, but for a different reason: Every type can be serialised over network or onto the disk, and when it's written to the disk it's usually in a format we can mmap() and access transparently. This is how q is also a database -- the data types q supports also includes tables.
k/q also has a built-in event loop that's not dissimilar to what you get when you run nodejs with the debugger port open except it's fast, and it's the regular way k/q processes communicate with each other.
What typically happens is that a table is designed for configuration, and we just expose it to the UI. Production environments are usually locked down so the only UI is to edit existing configuration parameters (and those are permissioned accordingly). These UI are typically quite general for any k/q data type, so they're quite rich and easy for people to use.
Then parts of the application interested in configuration just query the appropriate configuration table - this is only about 1000x faster than you would expect connecting to a remote database, and in many ways it's similar to a python application just storing its config in an sqlite database, except SQLite doesn't let you have a table as a data type so you can't put a table into another table, and you don't have tooling around comments and advice like you do with k/q UIs.
There are other places if you look carefully: Environments people have used (even beyond thirty years ago) that didn't have configuration "files", often had interesting and useful solutions to storing configuration. People tended to build configuration into part of their application, and so the legacy of that has tended to be excellent tooling instead of novel file formats.
[1]: http://www.shaffer-consulting.com/david/Seaside/Configuratio...
[2]: https://kx.com/
"From the point of view of a component, the configuration can be thought of as a Dictionary-like collection"
This is exactly what YAML and JSON provide too. Using a configuration file is a choice, so are parameters, and so are databases. I'm not really understanding what you're trying to get at?
The reason configuration file [formats] exist is because many programs are configurable and programmers are too lazy (or are not specified to, take your pick) to build a configuration tool [that has all their needs]. Configuration files are inferior in every way to an integrated and well-thought-out configuration process except that they may be easier to build and use in less ideal environments.
JSON is a fine format for interchange, and even persistence (i.e. to store configuration) but as a "configuration file" that people are expected to edit in their own way it is lacking, and that's why there are things like YAML and TOML and a million other things.
I recall the first time I saw YAML and all I could think to myself was that I have to learn yet another syntax. I find it far less readable than JSON or XML and made me pine for the latter.
YAML is bad.
Every YAML parser is a custom YAML parser.
https://matrix.yaml.io/valid.html
Is it TOML as the author seems to prefer at the end?
(More recently, I've come around to prefer plain environment variables for configuration, but that only works nicely when the amount of configuration is fairly limited, say 20 values instead of 1000 values.)
For my own use, I do prefer TOML.
More here : https://hitchdev.com/strictyaml/why-not/toml/
I have some projects where I'm frequently writing and midifying content that resembles the example here, and I use YAML there and plan to keep using YAML. For most other things, I'm just doing configuration, so I use TOML. No reason you need to stick to one or the other.
It's overly verbose, and hard to understand XML, it's no comments son, horrors of yaml or some okay format that doesn't have parsers for the languages (plural) you are using on your project.
I've also used the Python port without issues.
TOML, in my opinion, is like a weird mishmash of JSON, ini, and bashisms. Though I have worked with it a lot less than the other formats, so YMMV.
https://dhall-lang.org/
The biggest problem with config formats is they mislead users into thinking they understand the format. The user tries to edit it by hand, and chaos ensues. So only formats that are stupidly simple, or whose warts are already familiar and well documented, are good choices.
Apache had a great configuration format. Nothing else used it (that I knew of) but you could in theory implement "Apache configs" and then people'd just have to look up how to write those, which there's lots of examples of.
JSON and YAML and XML are data formats; they should only be written by machines, and read by humans. Same with protocols like HTTP, Telnet, FTP... You're not supposed to write it yourself, but it's readable to make troubleshooting easier.
Data formats are nice for expressing nested data structures, but then they don't (usually) support logical expressions; at that point you need a template/macro/programming language, and at that point you're writing code, which will need to be tested, and at that point you should just write modules and use a config format to give them arguments. Every complex tool goes through the same evolution.
If you care about your users, write a tool to generate configs based on a wizard. Good CLI tools do this, and it really makes life better. (It's also a great way to document all your config features in code, and test them)
(No, not YAML.)
Aside from lack of comments, the other major thing that can sometime make json a bad config choice is lack of multi-line strings.
https://json5.org/
The idea is to have 2 levels: a simple, minimal syntax/notation (think binary) called Tree Notation, and then have higher level grammars on top of that, called tree languages.
It works for encoding data and also for programming languages, regardless of paradigm.
https://github.com/treenotation/jtree
Here’s a BNf:
https://github.com/treenotation/jtree/issues/1
There’s a FAQ as well.
Docs needs work, in particular I’m hoping people will create their own explanations of the ideas in external places, as that might be a better way to understand it. Happy to provide help to anyone that is interested in that.
https://news.ycombinator.com/newsguidelines.html
Edit: we've had to ask you multiple times already not to be a jerk on HN. Would you please review the guidelines and take the spirit of this site more to heart?
Your impression is correct though, without highlighting it’s awful. Try the Tree Language designer app to see it with highlighting, type checking, autocomplete etc...there are interesting ways to accomplish everything without syntax, often not obvious. But tooling is essential to make it better than existing options. Help wanted!
You might end up with valid YAML, but you won't know until the YAML consumer barfs.
BTW, all of a sudden XML with DTDs are looking sane again :)
you can quickly tell if an xml document is malformed (good parsers will tipically point you to the un-closed tag).
Yaml on the other hand would probably load anyway, with the application receiving garbage data, potentially misdirecting the application behavior...
This refrain just cheeses me right off every time. Nothing is human readable! Everything requires a program to read it, because no human being can read states of charge or states of magnetic polarization directly.
What makes something 'human readable' or not is a software tool. Underlying that tool is a data format that the tool can accept and display. What everyone means when they try to sound smart by saying 'human readable' is just 'plain text.' In other words, they know where to find the dumbest possible reader/editor for it. Text editors are the dumbest possible editors because they cannot constrain edits to conform with the grammar of the interface language; they allow bugs at a point in the development process where it is trivial to disallow bugs, especially considering that interface languages should probably be, at most, regular languages.
I'll step off my soapbox, now.
Personally, I think most editors are rather specialized. They deduce the character sets, often add syntax highlighting and provides paging for very large files. High level strongly typed languages, such as modern C#, Java, Scala are designed to be used in an IDE. You could view and edit it, but it provides a difficult situation (not unlike editing XML by hand).
"Human readable" is very subjective. It depends on the person, the task at hand, the intended recipients, etc. etc.
Every veteran in the field knows that data and data structure are the primary enablers for almost any solution. Nevertheless, we have regressed in the last decades w.r.t. that. Had we had better structured (AST-driven) editors and context dependent representation, XML (or sth similar) might have taken off.
Personally, I blame the exponential increase in inflow of new developers. Nothing but reinventing the wheel without knowing history.
- Meaningful version control
- dump it to SQL and version control that, if you must use Git
- Not everyone likes the extra step.
- not everyone likes the hoops you need to jump through with the alternatives either! That’s why we’re discussing this :-)
This is simply wrong. There is nothing in the spec stating that.
Could you elaborate on this? I use Ansible daily and I've never had a problem with YAML once I took some time to understand it. What do you mean by broken parsers? I'm assuming that's something Ansible specific you are referring to.
And ansible's parser is broken, in more ways that I can remember (haven't been writing playbooks and stuff for a couple of months now). If you like pointless pain, try embedding ":" in task names for a demo (or one of other several "meta" characters: the colon is just the one that ends to recur most).
I will give a passing mention to the smug, vague error message "You have an error at position (somewhere in the middle of the file) It seems that you are missing... (something) we may be wrong (they almost always are) but it appears it begins in position (some position close to the first line)" that sets off a hunt for the missing brace/colon/space/whatever and makes me want to do stuff to the person who devised it.
This compounds with the confusion brought on weaving of yaml's and jinjia2 syntaxes and ansible's own flakiness on deciding what is evaluated when - which decides when and if a variable does indeed change, when does yes means "yes" rather than true or 1, but not '1' or "true" (try prompting the user for a boolean variable, and find yourself writing if ( (switch == "true") or (switch == "1")) in short order).
Pity that ansible is so damn convenient, or I would have ditched it long time ago for anything - bash included (OK, maybe not bash).
Instead just declare all config variables within code itself in a separate config class/module file, along with initialization to default values and provides dynamic getter/setter interface over a debug API (which can be enabled/disabled via a command line flag).
If you want, you can also provide a friendly cli tool to interact with the debug api. This tool could output help messages, show current config values - differentiate between default vs overwritten etc.
Of course, this can be written once as a utility library and cli and used consistently across all your programs.
1) it presents an obstacle to sharing config files in a multi-language environment.
2) writing user prefs in the host app’s native language is fraught with security issues
The first concern is admittedly niche, and both concerns are addressable with some care and thought, but they’re good to consider.
And we load the config by curling a JSON payload :) ?
Config files are fantastic. Trivial to read, write, copy, track in version control, diff, grep, generate with scripts, etc.
API-driven configuration has none of these properties.
Some Java application servers take this approach of API-driven configuration. It's an improvement over UI-driven configuration, which is what they had before. But it's still significantly worse than simple file-driven configuration.
If you want to provide a 'friendly' CLI tool, by all means do so - but provide a tool to interpret and generate config files, not something which replaces config files.
Code based configuration offers most of those benefits, just look at some of the suckless tools, if you've never used dwm or written a line of C I bet you can still guess how to configure some stuff here: https://git.suckless.org/dwm/file/config.def.h.html .
It doesn't work for all software of course, but for anything developer centric and/or anything with complex configuration done by experts it's a good choice, particularly for any software going down the path of creating a custom language (tmux, vim, etc). You get the power and flexibility of a full programming language, you get compile time config checks, you get excellent performance, you don't have to learn yet another config language and it's easy to write patches.
It's not great if you're targeting Joe Average, but neither is any configuration format, or any configuration at all.
If you need runtime reconfiguration in production, then it requires a runtime config management system tailored for operations folks with proper authn/authz, audit logs etc. This is a product by itself. The connection between your running program and the runtime configurator has to be intentional, secure etc.
IMO, config variables should have following bindings: 1. config variable in code. 2. program start env variable. 3. program start command argument flag. 4. runtime configuration.
All available configuration options are declared in the code. But not all configuration options should be accessible from #2 to #4. And the override preference order may not be same for all types of configuration variables.
Btw, this isn't related to Java or any particular programming language. I've seen this done in large C++ projects 20 years ago.
It just makes things run smooth in TypeScript because all the members and types can be statically analyzed giving you errors, auto completion and ability to jump to the definition when config files cannot do that but probably not good for large projects with multiple languages.
For small projects I'm just taking the benefit over small concerns.
So now no longer a YAML fan...
The big strike against YAML I see there is that it needs a good conformance test suite and implementations need to be tested against it. But that's not a problem with the format but a fairly easy to fix ecosystem problem.
But the syntax was valid, the parsers/validators would've been correct to accept it.
I have found myself using TOML more and more for configuration, though. It helps a lot with keeping things flat and easy to read. I'll still prefer YAML over JSON for human-writable files, but I'm starting to prefer TOML over YAML.
Boxes to check:
1. Self describing format
2. SAX-style parser available to C++
3. Easy for users to understand and ad-hoc parse using command-line tools
4. No document closing necessary, so appending is trivial
YAML looks pretty good:
protobuf is also an option: though I am unsure of how well its text serialization is supported.Any suggestions?
Ps. Thanks for (all the) fish, it's my daily driver shell and keeps me that much more sane c.f. the alternatives.
Having to close contexts is a VERY good 'sanity check' to see if something is malformed or not.
If appending is necessary make the parser handle multiple copies of the namespace and merge them upon output. Unknown keys and sections should also always be copied from input to output (this is how you embed comments).
To clarify the requirement, history could be a JSON array of objects:
To append an entry to this file and keep it valid, one must locate the closing square bracket and overwrite it. That work is what I hope to avoid.The object stream idea is apparently supported widely and seems pretty strong. Thanks for the suggestion!
Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.
A file already exists with this content:
[{"cmd": "git checkout", "when": 1234 }]
Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config.
[{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}]
A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...)
Better than nothing I guess, but I'd say just use a syntax that supports comments.
Try just using line-delimited JSON objects (http://jsonlines.org/). It ticks all of your boxes, especially 3: "jq -s '.cmd' fish_history | histogram".
Neither YAML or Protobufs are quite as easy as that.
All in all it's ridiculously simple, easy to parse in a variety of languages and each row is a single line that's simple to iteratively parse without loading the whole thing into memory.
Here's a proposal: use a Tree Language.
I created a demo for you called "Fished": https://github.com/breck7/fished.
Took me just a few minutes but already get type check, autocomplete, syntax highlighting, and more.
Tree Notation is early, and there will be kinks until the community is bigger, but I think it may be useful for you.
http://treenotation.org/designer/#grammar%0A%20fishedNode%0A...
Stumbled into the idea. Basically just brute forced it. Tried thousands of things, built a huge database of languages, and tried to keep it simple.
Basically, it’s a narrative discussion of how this solution came to be, the trade offs involved, and perhaps its relationships with prior art.
[0] https://docs.pylonsproject.org/projects/pyramid/en/1.10-bran...
Googling "Design defen{c,s}e" just got me a whole lot of military contractors.
It fulfills all of the requirements. There are several available C++ TOML parsers, including one from Boost.
[1] https://github.com/toml-lang/toml/issues/356
[2] https://github.com/toml-lang/toml#user-content-array-of-tabl...
Personally I really like the array of tables syntax. It is a little unintuitive but it's not difficult to remember. It's useful for fulfilling the OP's "No document closing necessary, so appending is trivial" requirement.
And if you don't want to use it, you can always use inline tables in an array, just like JSON.
I also wonder if you need a text format, or if SQLite or systemd's journal API would work.
A simplified subset, .syml would be a good idea.
The implementation is in Python.
[0] https://github.com/crdoconnor/strictyaml
For instance, the recent iMessage bugs that project zero announced were because NSCodable serialization tells the deserializer what class should instantiated. Followed by remote code execution (woo!)
Similar problems have occurred with java serialization over the years, the python serialisation thing (that silly name I can’t recall).
I was recently learning swift and was getting frustrated by the verbosity/work for deserialisaing abstract classes when I realized the clunkiness was due to a design that made the deserialise attacker specified objects basically impossible. Obviously you could engineer a solution that would be exploitable but there’s only so much a platform can do to stop developer mistakes.
I like the _idea_ of yaml, but: - it's overly complicated in the wrong ways - common/simple use cases aren't supported and require post processing (i.e. Merging block maps/arrays, string interpolation, etc)