As the article mentions, there's 'real' programming languages made for configuration, that solve at least some of the issues outlined, like Dhall, Cue, Jsonnet. After using both approaches a number of times (a general purpose language vs. one of the three), I'm imploring anyone trying to give this a shot to _not_ use a general purpose language.
For instance: I'm mostly familiar with Jsonnet, which has guarantees that make it much easier to use than a Real Language: no arbitrary file loads (paths must be static), no side effects, no ambient state (like env vars or Internet). Relative imports make it very easy to drop in the language somewhere in a repo and not have to worry about venvs/PYTHONPATH/system libraries... The interpreter being a single binary and a thin C library also simplifies integration quite a bit, compared to interfacing with full-blown interpreter/compiler installations.
The downside is, of course, that you must learn a new language. But we're all pretty good at this (unless you're one of those people obsessed with using JS/npm everywhere for some inexplicable reason). Also Jsonnet still has the pedigree of being a superset of JSON, which is painful (ie. magical int/float number type). But the alternatives (Dhall, Cue) fix that, and also provide a vastly better type system.
I've been using Jsonnet in production for a couple of years (after leaving Google where I Saw The Light with BCL), and couldn't be happier. The infrastructure of the Warsaw Hackerspace (a production k8s cluster) is all brought up using Nix and Jsonnet and is open for all to inspect [1].
And yes. Let's please stop using JSON and YAML for everything. Even Python is better than yet another YAML templating tool.
That is really cool - as chance would have it, I've just spent this weekend converting all my personal infra stuff to Jsonnet and I'm having a blast! This is some really nice inspiration.
"Old and quirky" is an understatement. It's a god-awful mess, with semantics so murky that people end up just copying patterns and hoping that it'll do what they want it to do. And it can't be killed, because every bit of those implementation-defined semantics are used by someone.
Borg Control Language, as others have said, Google's configuration language for "Borg", the precursor of Kubernetes. Both Jsonnet and Cue are descendants of BCL, developed by people who were previously involved in BCL development.
Jsonnet is a direct descendant of BCL, "BCL done right". Cue is an indirect descendant of BCL, you might say "BCL rethought".
I have a different approach. When I'm deciding about a new technology that we could add to our tech stack, I ask myself if it's really worth it. Adding a new tech is very expensive because it adds to the list of things that the developers have to be good at. I would avoid adding a configuration language and prefer using the same general purpose language that the rest of the software is written in.
At work, we have a C# ASP.NET Core application we build.
- We use Pulumi's C# API to deploy to the cloud. This lets us avoid yet another YAML based config format, and it lets us edit the deployment in a familiar language with a pretty simple API in our favorite IDE. (Link: https://www.pulumi.com/)
- We're going to move to Nuke instead of having Azure Pipelines YAML files. One less YAML file format to remember. (Link: https://www.nuke.build/)
- We're going to do benchmarks with BenchmarkDotNet, to keep our code in C#. (Link: https://benchmarkdotnet.org/)
- We're going to be adding Spark-based analytics, and I'm undecided about whether we should use Python or the C#. Python is so ubiquitous in analytics and there are so many libraries that it might be worth adding another language to our system. (Link: https://dotnet.microsoft.com/apps/data/spark)
Same here, but stuck with json/yaml, etc. What are you thoughts on starlark for configuration language? Imperative, but with lots of functional/declarative bits - like frozen vars, no recursion, sandboxing, execution in parallel... I've also looked at jsonnet, even contributed one or two patches to speed it up in C++, but still felt the evaluation time was higher (the "go" version is faster than C++, but they (last I've checked) used slightly different evaluation rules, also C++ would get slower in bigger app due to the overuse of dynamic_cast (at least that's what I've found)).
Yeah, I'm also trying to find the magical bcl/gcl - but what was so useful (when I worked at google) was the diff tool (was it written by an intern, it was amazingly useful) - e.g. your presubmit would tell you - how the end (evaluated) result would look like, as one change in one place might have quite the ripple effect.
> What are you thoughts on starlark for configuration language?
I haven't tried that yet. It might work. I mean, I've used starlark as _a_ configuration language (notably, in Bazel, or as a little prolog-standin for defining complex rules), but never really built an entire infrastructure on top of it. I would have to look into how it works my usecase (can't lie that jsonnet having an existing third-party kubernetes library was a big seller for me, even if I end up extending it). I do prefer, however, functional/lazily evaluated/declarative languages for configuration - while starlark has a very imperative taste to it.
> [...] but still felt the evaluation time was higher [...]
Yeah, that can be painful, but has not fully bitten me - my configs rarely get large enough to notice, and even if they do there is a large bottleneck somewhere else (ie. actually applying the config). Definitely also have it on my backlog to look into sending some patches for that.
> was the diff tool
There exists something like this, kind of, for when you're using jsonnet to emit k8s resources: `kubecfg diff`. But that's for diffing against prod, not an evaluation against a previous VCS tip. Definitely a tool that should exist, though :). I do think Dhall is a bit further down the road of having this sort of tooling actually exist upstream.
I'm not convinced - the ergonomics aren't great, it does not support CRDs, is mostly untyped, and you end up unnecessarily encapsulating everything with dozens of function parameters since you cannot easily merge nested data.
Starlark is great for configuring an imperative tool like Bazel which is only declarative at the top level, but is impractical for pure configuration data. Having the full power of a general purpose language isn't great for configuration - it quickly leads to lots of complexity at the wrong layer.
Cue has a radically different approach[1] that works very well in practice. The author of Cue, mplv, wrote the first version of bcl/gcl and Cue is the spiritual successor of sorts :-)
It's still early, but many people (including me) already use it in production. Their Slack is very friendly and helpful!
Cue gets a lot of things right (strong builtin validation, strong typing), I haven't explored it much, but I feel that a language that bans inheritance mixin-inheritence, and uses essentially typed structs that can have computed properties, and late-binding, gets most of what you want. Something like this could be put onto starlark/skycfg fairly easily, I think.
> no arbitrary file loads (paths must be static), no side effects, no ambient state (like env vars or Internet)
This all sounds like it should be a part of the sandbox, but it's not clear why it actually requires a whole new language. Many existing runtimes can be sandboxed in a similar manner. So why not just sandbox e.g. JS for JSON?
There are cool things somewhere in the middle now too, e.g Cue [1], which is really cool.
I was recently thinking that its a shame there isn't a cross language proto-lite kind of thing (I'd settle for json with structures and pointers) that has a great devx. The multi step transform that configs are right now (from the serialization representation to the code one) could be eliminated. IMO interop is one the most important things for configs, so I'd be hesitant to use a thing that needs a preinstalled runtime.
As we seem to be heading towards immutable-'everything' though, sometimes I wonder how valuable dynamic (in the sense of picked up at start) configuration is for backend services. It seems preferable to build everything, config included, into one single artifact. Potentially this gives more optimization opportunities as a lot branches could be completely eliminated (feature flags, etc.). No doubt the simplest way to build your config directly into your code is to just.. have it be code.
A potentially super annoying trade-off is that it's difficult to test minor configuration tweaks, and the annoyingness scales non-linearly with your build time.
And even more annoying if you would rather not restart your services unless absolutely necessary (ie. you're handling long-lived TCP connections that cannot be handed off gracefully). Not to mention that some kind of configuration tweaks (ie. ACL changes, quota changes, etc) might happen at a vastly different cadence than software rollouts.
> if you would rather not restart your services unless absolutely necessary (ie. you're handling long-lived TCP connections that cannot be handed off gracefully).
If you have no way to hand off your TCP connections gracefully, that is probably not a great position to be in because computers are unreliable and sometimes you need security updates. One approach I tried experimentally for a TCP server (but unfortunately did not have the opportunity to try in production) was to have a small server acting as the frontend that handled sessions and called into the real implementation which was essentially stateless. This small server could theoretically support self updating without dropping connections, if need be. (The main reason I was doing this was so that the rest of the stack could be dynamically-scaling based on Kubernetes, while this part remained relatively static.) I think anytime you have forced statefulness, it’s worth isolating as much as possible so it doesn’t constrain the design of the whole system.
(Even better, of course, is to just have a protocol where clients can gracefully handle draining and reconnect without side effects.)
> ACL changes, quota changes, etc
At a certain level I think these are runtime data and not configs. Aforementioned service also dealt with ACLs and configs and we were serving it through another service that persisted to a database layer. One cool thing about doing it as a service is that you can manage and scale it like any other service.
>I think that's the gist of the issue - sometimes it's difficult to tell which one is which :).
Absolutely... in fact, there's definitely stuff that I've had start as configuration and gradually turn into runtime data, and there are things I would call configuration that need to be dynamic at runtime too... I'm never 100% satisfied with the split.
Ideally this isn't how you achieve HA - machines are unreliable and therefore service instances are too. Absolute statelessness is the goal, but of course that's exceptionally hard to achieve in the real world and bopping live connections will always cause annoyances.
No, I can’t identify what’s wrong with it. Why am I spending what feels like 5 minutes studying a code sample to find a flipped ‘I’ and ‘L’?
Every time I read about dahl, I remember this and it sours my patience for the language.
It’s a good point, but there’s got to be a better way of communicating the value. Or, maybe just having a button that says “spoilers” and reveals the issue.
I couldn't figure it out either! But now that you tell me what's wrong, it looks completely obvious. And I agree that it turned me off on reading the rest of the page.
Bazel uses a subset of Python for describing build rules
Bazel's configuration language is called Starlark[0] (used to be Skylark). It's not a strict subset of Python. It has implementations in Java[1] and Go[2]. I haven't had a chance to use it yet, but it seems very useful as a general-purpose scripting language, especially for embedding into tools.
As someone already put it, 'Bazel is the worst system except all the others'. It has some warts (Python integration, non-hermetic-by-default rules, starlark structure javaesque complexity ...), but I still haven't far a good replacement for my use: a monorepo build system that has to touch a ton of different languages and can be run in CI without having to prebuild tons of fragile builder images.
It absolutely is, especially if you do C++ with it. So many bugs and missing features in the native rules and skylark api, it's not even close to a 1.0
I've had great success with the Go Starlark interpreter. Very easy to get started with. I map some of my internal structs to its dictionary type and let CLI callers filter my collections by providing predicates as Starlark expressions.
The mapping to and from Go types is a bit obtuse and could probably be smoothed over by reflection or an intermediate representation like JSON in many cases. It's not just data though: you can also send functions in both directions.
Embedding an interpreter for filtering vs. trying to compile all possible filtering needs into the binary greatly improved my program's power and shrunk the volume of application code. I recommend it.
Don't be alarmed by the Bazel connection: it's much easier to use and understand (and better documented) than Bazel itself. Although I am a lot better at Bazel now for having taken the time to understand Starlark.
All that said: if you find yourself needing to support logic and behavior in your configuration language, you may not actually have a good enough abstraction to justify a config-driven system. Take a hard look at whether it might be cleaner to write plain old code, or APIs for other people's code. A significant number of internal platforms at my company are unnecessarily obtuse because they try to put everything in boxes like this vs. being legitimately composable/programmable through ordinary software engineering tools and processes.
Bazel is the thing that I've missed after leaving Google, (lol "bazel menu"). Back then I was using it on the surface, there were no .bzl files, but internal python rules. I actually hated it when I started, because i was like - WTF is this shit, then I've learned.
My best example would be a CI system, given a CL (changelist) it can tell you exactly what targets need to get built. In our current CI - we have put extreme whitelsits - e.g. if you have touched this folder with C++ files, these are the .sln files to build. But can't move to bazel yet (mostly Windows shop). I actively use it for my home projects (linux + windows), but that's about it.
When you say "not a strict subset of Python", is that primarily the stdlib changes? Are there any syntactic differences that mean valid Starlark is not valid Python?
Yep yep, makes sense. I helped migrate a Python monorepo to Bazel, which involved an unfortunate amount of Starlark :-)
The mental lines between Starlark and Python are fully blurred at this point, but I've seen the "Starlark isn't Python" a few times this past week already.
I use schemy [0] for non trivial config tasks in .NET projects. LISPs in general are well suited for this.
Very small implementations are easily done and data is code, it’s pretty great.
Of course, if one has already access to a Python runtime that is probably fine as well.
I cannot disagree more. Configs are wonderful, as they follow the "rule of least power" with putting settings in a place without logic operations, variables, etc.
Yes, they are not general. Still, having one dedicated place for all constant switches is much better than constants being out in a dozen (or hundreds) of places all over code.
But that's the point of this. You can have a single high-level config that emits the same high-level settings onto multiple config 'facades'. For instance, a single jsonnet/python program will let you tweak a list of domains, and that will in turn create a facade (or generate a file) for nginx (to configure vhosts) and for a letsencrypt tool (to acquire TLS certs). Less repeating across files, less chance of making a mistake, and since this is a programming language, you can have a single codebase that performs facade generation, without having to go through a CM system.
But then your config files in Python also need tests. Are they still a config file then or config scripts? How do you limit what they can and can do? Easy to do with JSON or XML, harder with Python.
This is why I'm not a huge fan of using Python for this (see my other comment) - and would much rather recommend Dhall/Cue/Jsonnet. And you can (and definitely should) write tests for these, or even just assertions for generated facades. It's much better than the alternative, which is just testing in production/qa/canary.
I don't get it. If I get the choice between taking over a system where the configuration is self-tested, vs one that is not, I would much rather pick the first option?
You don't _have_ to test anything, just like in any software development. But now you can, and you can do it in a side-effect-free environment, on CI push, instead of realizing you made a typo when the CM kicks in, or worse, when your system starts crashlooping on startup because of a configuration issue.
If somebody hands over an application to me where configs have unit tests, this means to me that the application can fail catastrophically under certain (unwanted) configurations. And seeing how I might be the owner of said applications, chances are high that at some point I would need to change the configs.
I, personally, would rather prefer straightforward static configuration and very well written documentation on why certain configurations make more sense than others. That way, I know how and when to change something in the configs.
I absolutely do not share the view that "tests are documentation", because they do not communicate the most important thing in software engineering - the intent and reasoning behind a certain design.
> If somebody hands over an application to me where configs have unit tests, this means to me that the application can fail catastrophically under certain (unwanted) configurations
This would be a sign of a program that allowed input (configuration, in this case) that was unsafe. That is completely a different matter to whether a programing language is suitable for defining configuration. I suspect that many an application has been written that assumes that because it's consuming "static" configuration that is a simple text file, or a structured data file, that the input is completely safe. (SQL injection can come from any direction.)
I agree with you. The idea of programmable config languages is really, really misguided. Whenever somebody wants to use something like that, it's a sign for me that they haven't thought out their design properly and are content of letting abstractions leak into the config.
I have long argued that teams should write configs in one of the languages they already know and are using in the system.
This has many obvious benefits, including testing!
A key aspect is for the owner of the "config system" (the stuff that takes the output of each config program and applies it) to standardize the API for config generators. Inputs, outputs, runtime env, etc. Then let teams integrate with that API however they want.
This is how Lua got started - its predecessors started out as data description or configuration languages. The users felt it would be useful to have some forms of flow control, which led to the birth of Lua.
Indeed, I wrote a console-based mail-client inspired by mutt but with Lua for the configuration file.
The Lua code made it easy to configure things such as the path to sendmail, and your email address. But also writing functions to filter, colour, and display messages was a joy.
mutt is awesome, but the configuration language it uses is a mess of history. No loops and conditionals being somewhat limiting. People end up writing "profiles" by essentially reading the configuration values from a different init-file. Horrid.
I know the majority of developers hate XML config files, yet...
I try to make my apps depend on config files and manifests as little as possible, but DSL vs fully featured scripting languages has always been a big conundrum in software development. Is python a solution? the problem is that at some point you might be tempted to add json/yaml config files to your python config scripts...
about 5 minutes after you start using xml for configuration, you want to reference a bit of global information (environment variable, port, ....) and add variable substitution ${foo} as an extra feature.
Under no circumstance would I allow my projects to be configured by a full-blown programming language. There is no way to ensure that there are no regressions under different configurations. The amount of complexity introduced just for configuration is insane.
If you feel the need to put code in your runtime config, that means your design sucks.
But configuration is a complex task! At least from my experience. Limiting yourself to solving it with mediocre tools (for instance language that disallow writing tests or even comments!) is a footgun, where you disregard good engineering tools for stringly types, duplicated configuration.
We might have different target audiences of our work, but in my line configs should not require tests. Because their task is to adjust the functioning of an application. If they all of a sudden require tests, that means (at least to me) that it's not part of the configuration anymore, but a part of the application proper and should be treated as such.
For me the target is mostly verbose YAML configuration of existing software (which I can now emit from an in-memory object instead of templating and having to remember all the YAML quirks) and k8s manifests. Especially when I have to stamp the same piece of software multiple times in multiple deployments (environments/regions/etc).
For instance: I just brought up a new Factorio game server yesterday, just by adding a single line to my jsonnet configuration machinery: https://gerrit.hackerspace.pl/c/hscloud/+/246 . This in turn brought up a bunch of new k8s resources, including some persistent volumes, a deployment, and a loadbalancer service, after running a single `kubecfg update` command.
You could use the Maven approach: for whatever the declarative configuration isn't powerful enough to specify, you write a plugin. That plugin can then be developed and managed just like any other software project, rather than having to bring those tools into the configuration realm.
I guess it depends on how you look at it, but if the plugin is a self-contained module that can be managed independently from the app it's configuring, I wouldn't really say it's code that's "in" the configuration.
> ... but ad-hoc logic do exist.
True. I don't disagree, no tool is right for every job.
Personally, I group these use cases under "infrastructure provisioning". The use case here is completely different - for example, you won't have a non-developer modifying the configs. So in this case, automation is a good idea.
Configuration is complex if you use it for complex things. That's just one way of designing software, it's not necessary to make software so extremely configurable that you need a DSL to configure it. Even if you really feel you need that level of configurability, then the software could be designed as a library instead and users could consume it with whatever language they prefer (where bindings are available). That would basically be the same amount of complexity as a configuration DSL, except you could use all your standard project management tooling to manage it.
Should it be? If an application doesn't have configuration, then the logical thing to do would be hard code all the values it needs in the code itself. This could be done with a number of variables/constants defined in the entrypoint method for each value needed, or defining a data structure that contains all the values needed in the same method.
Reading a config file and serializing it into those variables or data structure shouldn't be a hard problem. Formats like YAML or json allow for deserailizing the string into a data structure (though you have to add your own validation). ini format files can be done the same way in a loop, but still require adding validation.
The validation could be based on data type or allowed values. While that's rather tedious, it's not complex.
Proof, no. But ensuring it in terms of manual labour is far easier with a static file, than debugging a convoluted script pretending to be a config file.
That’s why my personal preference is to keep the config file as code but to keep it as close to a static config as possible. The Turing completeness of the environment is there merely as an escape hatch, for the one branch you need to make.
If your tooling requires static configs, but you really need dynamism, you’re stuck with dynamically generated static configs, which are even worse than a dynamic config.
As I've mentioned elsewhere, I agree, but I think in an ideal world, there would be an explicit syntactic construct warning everyone where the program was using the escape hatch to leave the Turing-incomplete subset.
You could create and use an embedded DSL tailored specifically for your application. This is what Jenkins does with their "declarative pipeline": it's basically Groovy-based DSL, and if you need anything not covered by it you need to use `script { ... }` element.
The problem here is of course how not to make your DSL suck, which Jenkins' did not avoid. But if you're lucky, your language of choice already has a few libraries implementing embedded DSLs for configuration, and it's quite possible that at least one of them is actually usable.
If you just want to use the type system, why not just deserialize your JSON config into a Plain Old Object? At least in C#, you can mark a property as required and the serialization will fail if a property is missing or the wrong type.
Generate static configuration, effect only what's in the static configuration and persist it to version control. You get the ergonomics of a programming language while getting a diff if anything changes and get it code reviewed. Could even write unit tests if you start to get unintended changes.
That seems like an acceptable solution to this. I'd still prefer not to be in a situation, where configuration is so complex and dynamic, that I need config generation, but if one does land in such a situation, I think this is the most sane approach.
> Under no circumstance would I allow my projects to be configured by a full-blown programming language.
Well, the choice is Java, or Java and yaml.
> The amount of complexity introduced just for configuration is insane.
Again, it's Java, vs. Java + Spring + env + application.yaml + application-$profile.yaml (in my case usually).
Once config constants enter your language, they're type-checked. Tree-structures suddenly need to be well-formed. No more trying to remember if you force https-only with 'enforceHttps', 'enforceHTTPS', 'enforceTLS', 'enforceSSL'. (= True,= true, :true, :1, :on, = "on")
> no regressions under different configurations.
It's great for that! Instantiate three copies of your service from Main - say, one backed by a hashmap, one connecting to a live db, one connecting to your docker-compose db, etc. Run the the same set of commands (in the same language your tests are already written in) and assert the outputs.
I find general-purposeness a great idea if your specific config depends on something complex, e.g. the username/host/path where the code is running. That's quite hard to in a confgiuration-specific format, without anticipating this specific scenario in advance.
Then again, we have Chef (ruby DSL), and from experience I've found that they tend to turn into a mess that very easily can become hard to reason about. Although, I'm not sure if this is because of the design of Chef itself, or just simply because it uses a real language.
Makefiles aren't executable - they are fine. .bashrc is badly designed. Bash should have a proper non-executable config system. But... it's Bash. If you're expecting sane robust design from Bash you haven't been paying attention!
Naturally, trying to do anything even slightly complex with it quickly leads to an unreadable cacophony of dollar signs, backslashes, and soft and hard tabs (you need both!). Just like the modern YAML-based systems, Make’s language wasn’t originally intended to be a full scripting language, but had scripting bolted on because people needed it. If only it had chosen to embed a real programming language instead, you could have the functionality without sacrificing readability.
It's not an ad-hoc turing complete language, it is just a programming language. Makefiles are just programs. There are some strange implicits, but that isn't unique to the Make programming language.
It's Turing complete because all standard programming languages are Turing complete (modulo some technicalities which are not relevant here).
It's ad-hoc because it was built gradually on top of the original Unix make, which supported rules and variables but not any of the more advanced features. This history explains many of its quirks and relatively extreme limitations as a programming language, such as:
- Expressions must be all on one line, except within `define` blocks or if you use backslashes to join lines together (which tends to result in a lot of backslashes).
- Hard tabs are required to introduce commands in rules but are (mostly) banned outside of them, so if you want any kind of indentation for an if block or user-defined function, you need to use spaces instead.
- User-defined functions. Calling them is merely more verbose than calling builtin functions (you need to invoke the `call` builtin function). Defining them is worse: function parameters are numbered rather than named, similar to a shell, but in a shell you can reassign them to named variables, whereas Make's expansion rules make that awkward to achieve.
- And of course, no types other than strings (a limitation partly shared by shells, but most shells do have arrays, even if they're awkward to use).
Admittedly, most of that gradual evolution happened a long time ago, and GNU Make has been relatively stable as a language since then. It's not "ad-hoc" in the sense that it's constantly changing or ill-defined. But that stability also means that it never outgrew the limitations of its original design.
Some research I did for fun:
The oldest version of GNU Make I can find [1], from 1988, already had a handful of functions, including `foreach`, `filter`, `patsubst`, etc., as well as support for multi-line definitions (`define`/`endef`). `call`, on the other hand, didn't appear until 1999. Amusingly, its initial implementation came with a comment [2] bemoaning the lack of "nested lists and quotes", and even semi-seriously proposing that GNU Guile (Lisp implementation) be integrated into Make, in order to let Makefile authors use a 'real' programming language. No fewer than 13 years later, that proposal actually became a reality; unfortunately, it was an optional feature which distributions tended to leave disabled, ergo Makefile authors could not rely on it being present, so the feature has seen approximately zero use.
I mean the "rc" in those names stands for "run command". To me they are initialization scripts.
Also, Bash is a scripting environment (sort of), and VimL is a solid part of vim. These are scenarios, where you don't add undue complexity or bugginess to the software. But personally, I wouldn't kid myself and pretend that I can write applications as solid as bash or vim.
Makefiles are just bash scripts with dependency resolutions. So again, scripts and not configuration.
Just because Django is popular, doesn't mean that companies using it know what they are doing, or have weighed all pros and cons. I've seen some outright tragic Django setups with major companies.
> I've seen some outright tragic Django setups with major companies.
Well, the same thing can be said about pretty much anything. I've seen good and bad Django setups, so what?. The point is configuration as code works well if you know what you are doing. A config minimal language (toml, yaml, etc) may also work well as long as you don't need something that's not supported by it, and as long as you don't need to debug it.
> If you feel the need to put code in your runtime config, that means your design sucks.
I would look at Terraform and beg to differ - they started with a simple stringly typed language and had to build a typed language. Still no support for conditionals, so everyone has to abuse the only bit of control flow on offer - ternary variables and counts.
CircleCI is just as bad - it's a nice yaml format until you want a conditional step in workflow that's conditional on something other than the branch name.
It's very risky to settle on a static config at design time since the users will find a need. How would you cater to that user without offering a programming interface to configure it?
The author mentions XMonad, which does precisely that. However, you can update XMonad without "restarting" the program. When you update the config, which is just a haskell program - the config is recompiled and launched. The state of your current session is then passed over to the new config process, and the old one config process is discarded.
People often forget that formats such as JSON and XML is not meant to be hand written. It's a serialization format. It's meant to be parsed, while still human readable. It's very nice to have them human readable to make it easier to debug, and make quick changes. But you should probably use a tool to edit them.
I think not containing any logic is a useful feature for config files. Logic adds complexity and potential for bugs and vulnerabilities - I actually like that JSON forces you to use simple primitives as config variables. I hate it when config files start containing function definitions; you can't pass those around to different processes.
I'm generally against the idea of complex configs. Configs should be simple. Sometimes that requires extra planning or thinking from developers. Every time I've allowed logic in config files, I've regretted it.
The main annoyance about JSON is lack of commenting ability. This is a point I agree with but not a big deal IMO, it's still better than all other alternatives.
JSON is ideal in terms of being both machine-readable and human-readable. Some people would argue that YAML is more human-readable, but it's definitely less machine-readable. With YAML, there are too many situations where some random code somewhere will remove all the new lines and tabs (for whatever reason) and mess everything up. JSON is resilient to machine sanitization. JSON is simple, robust and readable.
I occasionally do this for python-only projects or for Makefiles, but (especially in the latter case) it's quite fragile.
I usually use libconfig for configuration, which is better than json and yaml and has implementations in many languages (although I'm not aware of any javascript implementation).
These days, using a statically typed language for configuration makes a lot of sense. You get stronger typing and with type inference not necessarily a lot more verbosity.
Kotlin and kotlin script seems to work reasonably well for Gradle. Gradle unfortunately has a bit of groovy legacy, which makes it a bit convoluted for some things. But Kotlin DSLs are quite nice for configuring things and it removes a lot of ambiguity when your editor can tell you a property doesn't exist or that a list instead of a string is expected. or help you autocomplete things that are allowed.
Basically the ultimate in configuration languages is something that has a minimal syntax but provides enough structure for tooling to help you validate things, provide autocomplete and other help, etc. There are now several languages emerging that have both static typing and some level of support for internal DSLs. Ruby sort of pioneered a lot of configuration DSLs back in the day but since it was dynamically typed, tool support for these DSLs was a lot harder. With languages like Kotlin, Swift, C#, Rust, etc. you get enough expressiveness that you can do similar internal DSLs but with the advantage of more robust tooling support.
This is a good example of a structured approach, with the caveat that there are multiple phases.
With gradle, there's three big phases: initialization, configuration, then execution. (See https://docs.gradle.org/current/userguide/build_lifecycle.ht...). I'd think you'd need to do something similar with any structured configuration DSL - have distinct phases where you may "init, plan, apply", etc.
This, it's a pretty complex problem to have. But that complexity may be worth it as part of a "developer productivity engineering" effort, where you're really looking to use data to minimize the time between a configuration change, and the time it's applied in a runtime environment. Doing that is a hard problem.
Agreed, but the best combination is type-safe and non-turing complete, ideally with restricted effects. [Dhall](https://github.com/dhall-lang/dhall-lang) gets this so right that I'm increasingly shocked that people never seem to have heard of it.
The only thing I don't like with Kotlin is the lack of a decent syntax for list and map literals. Unfortunately it seems to have resolved in a dead end as far as implementation into the language goes. It's still a nice language but quite awkward and ugly in the context of expressing a lot of literal data.
There was a time when I thought you want a declarative language for your build system. After having used Rust, which allows you to use Rust in your build scripts I think that that is the way forward. You can say generate game assets in your build script, access network, use third party crates etc etc.
Declarative languages are a work around for bad APIs.
I did this with a server-side TypeScript project I built last year. The biggest advantage I noticed (and it's mentioned in the article): validation (i.e. type-safety).
Specifically, the project uses TypeORM and it was nice to be able to import the types for the DB connection options to make sure I had all of the necessary properties.
I was using ts-node to run server-side without compilation, which made importing an `app.config.ts` file quite elegant, IMO.
It's a nice balance, I think - most of the time you can treat it as a normal config file, but if you need to drop into some programmatic stuff you can. I've done this in my own dotfiles to add custom package definitions before contributing them upstream, for example: https://github.com/jfrederickson/dotfiles/blob/master/guix/g...
Using json or yaml for your languages config is nice for the application but it's not great for for the developer especially if they want to reuse values or settings across multiple applications. The solution then is to use a real programming language to generate that config. This is why languages like Dhall, JSonnet, and Cuelang exist. It's why I built my own toy language UCG to explore the space. But you could just as easily use python, javascript, or some other language too.
If you think that's punishment then you haven't suffered enough. I've seen enough custom written config formats in my life that choosing a widely supported format like YAML or JSON is always the right choice. YAML may have warts but you can learn them in 10 minutes and you're done. Those custom configuration formats don't even have a name or documented syntax.
Not a popular answer, but XML is mature, widely supported by many programming languages, and addresses all of the frustration criteria mentioned in the article:
- doesn't have comments
XML uses the same <!-- stuff here --> SGML-style comments that HTML does
You can technically put logic in XML using XSLT (https://www.w3.org/TR/2017/REC-xslt-30-20170608/), but I would ask if one SHOULD put logic into your configuration. I personally would recommend against it if at all possible. Keep configuration as declarative as possible and put the logic in your program. It's better to have a configuration logic module or use dependency injection than put logic in your config.
- Programming language constructs are reinvented from scratch
Again, XSLT could be used, but should your configuration really need to be Turing-complete?
Granted, XML is complicated and not without criticisms, but it is mature, standardized, and well-supported. Python has great XML support: https://docs.python.org/3/library/xml.html
XML can be used portably between different platforms and programming languages depending on how the data is represented. XML Schema defines a way to describe data types that is flexible and interoperabile: https://www.w3.org/TR/xmlschema-2/
Again, I know it's not the popular answer, but I was surprised that it was not even mentioned in the article.
The main reason I'm not using XML seriously is that it only really has a single native, scalar type: string. Sure, you can schema validate away a large part of this problem, but it's still somewhat of a wart.
With XML and other "fully qualified" syntaxes the named closing tag makes it easy to visually see where a block starts and stops. Humans parse the text too. This begs the question: Do you read config more or do you write config more?
Braces and parens all look the same, so its harder to visually match them. "But wait," you may say, "my editor / IDE does brace and paren matching so I can easily see and jump to the corresponding brace or paren." True, but doesn't your editor / IDE also support or have plugins for automatic HTML / XML end tag insertion?
Personally, I am equally comfortable with XML and JSON style configuration, but I find the brace-and-quoteless style unsettling because I believe that whitespace is a poor choice for delimiting structure. Your opinion may vary and that is fine. Do what works best for you.
In my opinion one of the weaknesses of XML is that it gives so many ways to express the same data, in attributes, as nested elements, with CDATA, and probably different other ways. And often it is a matter of taste. So in the end you come up with so many approaches to express similar structures.
>> So in the end you come up with so many approaches to express similar structures.
Expressiveness and flexibility are good so that you can define configuration to meet the needs of the system or application you are building.
XML Schema, Relax NG, etc. can be used to specify how the XML configuration should be structured, limits on data types, required versus optional configuration items.
As I said before, XML gets a lot of flack for being verbose, ugly, and complicated, but it is mature, widely-supported, and might be worth considering depending on your needs.
Not forcing them to learn a programming language, but instead forcing them to learn a half-assed quirky ham-strung config language that's just as complex as a programming language's syntax, and as hard to learn as a programming language, but isn't useful for anything else, or widely supported by any other tools, or well documented, or well designed, certainly isn't doing them any favors, or respecting their time and effort.
Case in point, PHP's "Smarty" templating engine. Because lord forbid people learn to use PHP as a templating language (which is ALREADY IS), when instead they can learn to use an ad hoc, informally-specified, bug-ridden, slow implementation of half of PHP, which is implemented in PHP, and dynamically generates PHP code.
>This is where folks will start to get technical about why their code isn't spaghetti code.
>Oh no! Not me because I use (Django|Rails|Haskell|Node.js|Symfony|CakePHP|Tornado|Plone) and the superior templating language (Django Template|ERB|Mustache|PHP|TAL|Smarty) doesn't allow business logic in templates and enforces a separation of concerns
>-You, the reader, December 2011
>The problem is that in every single one of these circumstances, the templating engine has 2 primary responsibilities:
>The fragmentification and inclusion of common interface assets (to avoid duplication and maintaining the same code in multiple files) - eg. headers, footers, sidebars and the like
>The output of dynamic content
>The output of dynamic content requires logic (if/else, loops being about the bare minimum).
>The creators of templating languages have, to varying degress, availed themselves of a false dichotomy, namely that there exists "presentation" logic and "business" logic, the former of which belongs in templates and the latter of which has absolutely no place in templates and must be banished immediately.
>I say false dichotomy because this is an arbitrary separation - a choice made by the person who created the templating engine and is completely subjective.
>Which brings me right back to my chief criticism of smarty which to this day it’s adherents can’t answer – what the Hell is the point? [...]
>I hated smarty from the day I tried it and I was always wondering why would anyone use it since you’re forced to learn some weird syntax that’s so similar to php - so why not use php from the start?
The trick is to get them programming without them even realizing they're doing it.
There's examples of this working effectively: millions of people can use excel to perform computations on groups of cells - the formulae they're entering include basic programming constructs like assignment, selection, loops, etc - yet it's not presented as a "programming system", which otherwise seems to put people off.
I recall reading an anecdote about the TECO editor (or some other emacs predecessor). Where the editor was being used by secretaries and other non-programmers, but they'd have no problems configuring it with the documentation they had available - they were never told that they were actually programming when they were doing so.
Perhaps it's a bit much to give the user a full-blown programming language for their configuration and expect them to have no problems, but it seems like limited programming concepts can be learned by just about anyone if presented in the right way.
I agree. I have seen incredible things done in Excel in the most elegant ways, by people with zero programming, math or computer backgrounds.
And as you said, presentation is key. Presented as "use these tools to set this up" things could go well. Presented as "write this in python" it may not go at all.
352 comments
[ 3.3 ms ] story [ 255 ms ] threadFor instance: I'm mostly familiar with Jsonnet, which has guarantees that make it much easier to use than a Real Language: no arbitrary file loads (paths must be static), no side effects, no ambient state (like env vars or Internet). Relative imports make it very easy to drop in the language somewhere in a repo and not have to worry about venvs/PYTHONPATH/system libraries... The interpreter being a single binary and a thin C library also simplifies integration quite a bit, compared to interfacing with full-blown interpreter/compiler installations.
The downside is, of course, that you must learn a new language. But we're all pretty good at this (unless you're one of those people obsessed with using JS/npm everywhere for some inexplicable reason). Also Jsonnet still has the pedigree of being a superset of JSON, which is painful (ie. magical int/float number type). But the alternatives (Dhall, Cue) fix that, and also provide a vastly better type system.
I've been using Jsonnet in production for a couple of years (after leaving Google where I Saw The Light with BCL), and couldn't be happier. The infrastructure of the Warsaw Hackerspace (a production k8s cluster) is all brought up using Nix and Jsonnet and is open for all to inspect [1].
And yes. Let's please stop using JSON and YAML for everything. Even Python is better than yet another YAML templating tool.
[1] - https://gerrit.hackerspace.pl/plugins/gitiles/hscloud/+/refs...
Edit: here's a thesis that works with a related language, GCL, with code samples: https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1....
I have a different approach. When I'm deciding about a new technology that we could add to our tech stack, I ask myself if it's really worth it. Adding a new tech is very expensive because it adds to the list of things that the developers have to be good at. I would avoid adding a configuration language and prefer using the same general purpose language that the rest of the software is written in.
At work, we have a C# ASP.NET Core application we build.
- We use Pulumi's C# API to deploy to the cloud. This lets us avoid yet another YAML based config format, and it lets us edit the deployment in a familiar language with a pretty simple API in our favorite IDE. (Link: https://www.pulumi.com/)
- We're going to move to Nuke instead of having Azure Pipelines YAML files. One less YAML file format to remember. (Link: https://www.nuke.build/)
- We're going to do benchmarks with BenchmarkDotNet, to keep our code in C#. (Link: https://benchmarkdotnet.org/)
- We're going to be adding Spark-based analytics, and I'm undecided about whether we should use Python or the C#. Python is so ubiquitous in analytics and there are so many libraries that it might be worth adding another language to our system. (Link: https://dotnet.microsoft.com/apps/data/spark)
EDIT: parent originally asked 'we use Pulumi, have you heard of it?'
Yeah, I'm also trying to find the magical bcl/gcl - but what was so useful (when I worked at google) was the diff tool (was it written by an intern, it was amazingly useful) - e.g. your presubmit would tell you - how the end (evaluated) result would look like, as one change in one place might have quite the ripple effect.
I haven't tried that yet. It might work. I mean, I've used starlark as _a_ configuration language (notably, in Bazel, or as a little prolog-standin for defining complex rules), but never really built an entire infrastructure on top of it. I would have to look into how it works my usecase (can't lie that jsonnet having an existing third-party kubernetes library was a big seller for me, even if I end up extending it). I do prefer, however, functional/lazily evaluated/declarative languages for configuration - while starlark has a very imperative taste to it.
> [...] but still felt the evaluation time was higher [...]
Yeah, that can be painful, but has not fully bitten me - my configs rarely get large enough to notice, and even if they do there is a large bottleneck somewhere else (ie. actually applying the config). Definitely also have it on my backlog to look into sending some patches for that.
> was the diff tool
There exists something like this, kind of, for when you're using jsonnet to emit k8s resources: `kubecfg diff`. But that's for diffing against prod, not an evaluation against a previous VCS tip. Definitely a tool that should exist, though :). I do think Dhall is a bit further down the road of having this sort of tooling actually exist upstream.
I'm not convinced - the ergonomics aren't great, it does not support CRDs, is mostly untyped, and you end up unnecessarily encapsulating everything with dozens of function parameters since you cannot easily merge nested data.
Starlark is great for configuring an imperative tool like Bazel which is only declarative at the top level, but is impractical for pure configuration data. Having the full power of a general purpose language isn't great for configuration - it quickly leads to lots of complexity at the wrong layer.
Cue has a radically different approach[1] that works very well in practice. The author of Cue, mplv, wrote the first version of bcl/gcl and Cue is the spiritual successor of sorts :-)
It's still early, but many people (including me) already use it in production. Their Slack is very friendly and helpful!
[1]: https://cuelang.org/docs/usecases/configuration/
This all sounds like it should be a part of the sandbox, but it's not clear why it actually requires a whole new language. Many existing runtimes can be sandboxed in a similar manner. So why not just sandbox e.g. JS for JSON?
I was recently thinking that its a shame there isn't a cross language proto-lite kind of thing (I'd settle for json with structures and pointers) that has a great devx. The multi step transform that configs are right now (from the serialization representation to the code one) could be eliminated. IMO interop is one the most important things for configs, so I'd be hesitant to use a thing that needs a preinstalled runtime.
[1] https://cuelang.org/
As we seem to be heading towards immutable-'everything' though, sometimes I wonder how valuable dynamic (in the sense of picked up at start) configuration is for backend services. It seems preferable to build everything, config included, into one single artifact. Potentially this gives more optimization opportunities as a lot branches could be completely eliminated (feature flags, etc.). No doubt the simplest way to build your config directly into your code is to just.. have it be code.
A potentially super annoying trade-off is that it's difficult to test minor configuration tweaks, and the annoyingness scales non-linearly with your build time.
If you have no way to hand off your TCP connections gracefully, that is probably not a great position to be in because computers are unreliable and sometimes you need security updates. One approach I tried experimentally for a TCP server (but unfortunately did not have the opportunity to try in production) was to have a small server acting as the frontend that handled sessions and called into the real implementation which was essentially stateless. This small server could theoretically support self updating without dropping connections, if need be. (The main reason I was doing this was so that the rest of the stack could be dynamically-scaling based on Kubernetes, while this part remained relatively static.) I think anytime you have forced statefulness, it’s worth isolating as much as possible so it doesn’t constrain the design of the whole system.
(Even better, of course, is to just have a protocol where clients can gracefully handle draining and reconnect without side effects.)
> ACL changes, quota changes, etc
At a certain level I think these are runtime data and not configs. Aforementioned service also dealt with ACLs and configs and we were serving it through another service that persisted to a database layer. One cool thing about doing it as a service is that you can manage and scale it like any other service.
I think that's the gist of the issue - sometimes it's difficult to tell which one is which :).
And I agree with the 'TCP connection in thin server' approach. Thankfully, I only have to deal with two protocols where this is an issue: IRC and BGP.
Absolutely... in fact, there's definitely stuff that I've had start as configuration and gradually turn into runtime data, and there are things I would call configuration that need to be dynamic at runtime too... I'm never 100% satisfied with the split.
No, I can’t identify what’s wrong with it. Why am I spending what feels like 5 minutes studying a code sample to find a flipped ‘I’ and ‘L’?
Every time I read about dahl, I remember this and it sours my patience for the language.
It’s a good point, but there’s got to be a better way of communicating the value. Or, maybe just having a button that says “spoilers” and reveals the issue.
Bazel's configuration language is called Starlark[0] (used to be Skylark). It's not a strict subset of Python. It has implementations in Java[1] and Go[2]. I haven't had a chance to use it yet, but it seems very useful as a general-purpose scripting language, especially for embedding into tools.
[0] https://docs.bazel.build/versions/master/skylark/language.ht...
[1] https://github.com/bazelbuild/bazel/tree/master/src/main/jav...
[2] https://github.com/google/starlark-go
The mapping to and from Go types is a bit obtuse and could probably be smoothed over by reflection or an intermediate representation like JSON in many cases. It's not just data though: you can also send functions in both directions.
Embedding an interpreter for filtering vs. trying to compile all possible filtering needs into the binary greatly improved my program's power and shrunk the volume of application code. I recommend it.
Don't be alarmed by the Bazel connection: it's much easier to use and understand (and better documented) than Bazel itself. Although I am a lot better at Bazel now for having taken the time to understand Starlark.
All that said: if you find yourself needing to support logic and behavior in your configuration language, you may not actually have a good enough abstraction to justify a config-driven system. Take a hard look at whether it might be cleaner to write plain old code, or APIs for other people's code. A significant number of internal platforms at my company are unnecessarily obtuse because they try to put everything in boxes like this vs. being legitimately composable/programmable through ordinary software engineering tools and processes.
My best example would be a CI system, given a CL (changelist) it can tell you exactly what targets need to get built. In our current CI - we have put extreme whitelsits - e.g. if you have touched this folder with C++ files, these are the .sln files to build. But can't move to bazel yet (mostly Windows shop). I actively use it for my home projects (linux + windows), but that's about it.
`load` does magic, so that `load('//path', 'name')` puts `name` into the file scope, which is possible in pure python but ewww.
Containers are immutable, so they can go in default values, which is good since a lot of bzl and BUILD relies on default arguments for brevity.
The mental lines between Starlark and Python are fully blurred at this point, but I've seen the "Starlark isn't Python" a few times this past week already.
Thanks for the elaboration :)
I guess my point is it's so similar to Python that in most cases I don't even notice.
Of course, if one has already access to a Python runtime that is probably fine as well.
[0] https://github.com/microsoft/schemy
Yes, they are not general. Still, having one dedicated place for all constant switches is much better than constants being out in a dozen (or hundreds) of places all over code.
You don't _have_ to test anything, just like in any software development. But now you can, and you can do it in a side-effect-free environment, on CI push, instead of realizing you made a typo when the CM kicks in, or worse, when your system starts crashlooping on startup because of a configuration issue.
I, personally, would rather prefer straightforward static configuration and very well written documentation on why certain configurations make more sense than others. That way, I know how and when to change something in the configs.
I absolutely do not share the view that "tests are documentation", because they do not communicate the most important thing in software engineering - the intent and reasoning behind a certain design.
This would be a sign of a program that allowed input (configuration, in this case) that was unsafe. That is completely a different matter to whether a programing language is suitable for defining configuration. I suspect that many an application has been written that assumes that because it's consuming "static" configuration that is a simple text file, or a structured data file, that the input is completely safe. (SQL injection can come from any direction.)
I do not think I would use this author's suggestion though.
This has many obvious benefits, including testing!
A key aspect is for the owner of the "config system" (the stuff that takes the output of each config program and applies it) to standardize the API for config generators. Inputs, outputs, runtime env, etc. Then let teams integrate with that API however they want.
The whole story can be found here: https://www.lua.org/history.html
The Lua code made it easy to configure things such as the path to sendmail, and your email address. But also writing functions to filter, colour, and display messages was a joy.
mutt is awesome, but the configuration language it uses is a mess of history. No loops and conditionals being somewhat limiting. People end up writing "profiles" by essentially reading the configuration values from a different init-file. Horrid.
I know the majority of developers hate XML config files, yet...
I try to make my apps depend on config files and manifests as little as possible, but DSL vs fully featured scripting languages has always been a big conundrum in software development. Is python a solution? the problem is that at some point you might be tempted to add json/yaml config files to your python config scripts...
If you feel the need to put code in your runtime config, that means your design sucks.
For instance: I just brought up a new Factorio game server yesterday, just by adding a single line to my jsonnet configuration machinery: https://gerrit.hackerspace.pl/c/hscloud/+/246 . This in turn brought up a bunch of new k8s resources, including some persistent volumes, a deployment, and a loadbalancer service, after running a single `kubecfg update` command.
In your scenario, it's more like provisioning of infrastructure and in that case - configuration is the code, so I can see where you are coming from.
I mostly write financial software so the use case is usually closer to classical single machine applications. And in that case, I prefer simple TOML.
edit: Now I understand that being a plugin it must have proper abstraction, but ad-hoc logic do exist.
> ... but ad-hoc logic do exist.
True. I don't disagree, no tool is right for every job.
Should it be? If an application doesn't have configuration, then the logical thing to do would be hard code all the values it needs in the code itself. This could be done with a number of variables/constants defined in the entrypoint method for each value needed, or defining a data structure that contains all the values needed in the same method.
Reading a config file and serializing it into those variables or data structure shouldn't be a hard problem. Formats like YAML or json allow for deserailizing the string into a data structure (though you have to add your own validation). ini format files can be done the same way in a loop, but still require adding validation.
The validation could be based on data type or allowed values. While that's rather tedious, it's not complex.
No, but static config files won't give you that proof either. Unless I'm misunderstanding your point?
At least if I write build scripts in e.g. ts-node I can use the type system to provide some modicum of safety, which seems like a plus.
If your tooling requires static configs, but you really need dynamism, you’re stuck with dynamically generated static configs, which are even worse than a dynamic config.
The problem here is of course how not to make your DSL suck, which Jenkins' did not avoid. But if you're lucky, your language of choice already has a few libraries implementing embedded DSLs for configuration, and it's quite possible that at least one of them is actually usable.
Well, the choice is Java, or Java and yaml.
> The amount of complexity introduced just for configuration is insane.
Again, it's Java, vs. Java + Spring + env + application.yaml + application-$profile.yaml (in my case usually).
Once config constants enter your language, they're type-checked. Tree-structures suddenly need to be well-formed. No more trying to remember if you force https-only with 'enforceHttps', 'enforceHTTPS', 'enforceTLS', 'enforceSSL'. (= True,= true, :true, :1, :on, = "on")
> no regressions under different configurations.
It's great for that! Instantiate three copies of your service from Main - say, one backed by a hashmap, one connecting to a live db, one connecting to your docker-compose db, etc. Run the the same set of commands (in the same language your tests are already written in) and assert the outputs.
I don't want to do step 2 unless I can do step 3.
It just so happens that type-checking happens during recompilation, and testing happens after.
See ansible and the YAML config files.
In this case, a real language would have been a better choice.
Also, sometimes you need to generate config files. In that case, templating is not always a good choice.
https://www.gnu.org/software/make/manual/html_node/Functions...
Naturally, trying to do anything even slightly complex with it quickly leads to an unreadable cacophony of dollar signs, backslashes, and soft and hard tabs (you need both!). Just like the modern YAML-based systems, Make’s language wasn’t originally intended to be a full scripting language, but had scripting bolted on because people needed it. If only it had chosen to embed a real programming language instead, you could have the functionality without sacrificing readability.
It's ad-hoc because it was built gradually on top of the original Unix make, which supported rules and variables but not any of the more advanced features. This history explains many of its quirks and relatively extreme limitations as a programming language, such as:
- Expressions must be all on one line, except within `define` blocks or if you use backslashes to join lines together (which tends to result in a lot of backslashes).
- Hard tabs are required to introduce commands in rules but are (mostly) banned outside of them, so if you want any kind of indentation for an if block or user-defined function, you need to use spaces instead.
- User-defined functions. Calling them is merely more verbose than calling builtin functions (you need to invoke the `call` builtin function). Defining them is worse: function parameters are numbered rather than named, similar to a shell, but in a shell you can reassign them to named variables, whereas Make's expansion rules make that awkward to achieve.
- And of course, no types other than strings (a limitation partly shared by shells, but most shells do have arrays, even if they're awkward to use).
Admittedly, most of that gradual evolution happened a long time ago, and GNU Make has been relatively stable as a language since then. It's not "ad-hoc" in the sense that it's constantly changing or ill-defined. But that stability also means that it never outgrew the limitations of its original design.
Some research I did for fun:
The oldest version of GNU Make I can find [1], from 1988, already had a handful of functions, including `foreach`, `filter`, `patsubst`, etc., as well as support for multi-line definitions (`define`/`endef`). `call`, on the other hand, didn't appear until 1999. Amusingly, its initial implementation came with a comment [2] bemoaning the lack of "nested lists and quotes", and even semi-seriously proposing that GNU Guile (Lisp implementation) be integrated into Make, in order to let Makefile authors use a 'real' programming language. No fewer than 13 years later, that proposal actually became a reality; unfortunately, it was an optional feature which distributions tended to leave disabled, ergo Makefile authors could not rely on it being present, so the feature has seen approximately zero use.
[1] within https://gcc.gnu.org/pub/binutils/old-releases/binutils-1988-...
[2] https://github.com/mirror/make/blob/c4353af3f9b143b213644c09...
Also, Bash is a scripting environment (sort of), and VimL is a solid part of vim. These are scenarios, where you don't add undue complexity or bugginess to the software. But personally, I wouldn't kid myself and pretend that I can write applications as solid as bash or vim.
Makefiles are just bash scripts with dependency resolutions. So again, scripts and not configuration.
Well, the same thing can be said about pretty much anything. I've seen good and bad Django setups, so what?. The point is configuration as code works well if you know what you are doing. A config minimal language (toml, yaml, etc) may also work well as long as you don't need something that's not supported by it, and as long as you don't need to debug it.
If you are willing to accept the defaults you aren't required to write any configuration.
Hello from every plugin system ever. True fact: not all configuration is basic.
And usually these things are implemented with way more forethought.
I actually agree with you. Elsewhere in the comments I say that the OP article failed to name this properly: https://news.ycombinator.com/item?id=22788696
But it is what the article is really discussing.
I would look at Terraform and beg to differ - they started with a simple stringly typed language and had to build a typed language. Still no support for conditionals, so everyone has to abuse the only bit of control flow on offer - ternary variables and counts.
CircleCI is just as bad - it's a nice yaml format until you want a conditional step in workflow that's conditional on something other than the branch name.
It's very risky to settle on a static config at design time since the users will find a need. How would you cater to that user without offering a programming interface to configure it?
For example if you set a path to Stack as a script shebang, you can write interpreted-mode shell scripts in Haskell.
I'm generally against the idea of complex configs. Configs should be simple. Sometimes that requires extra planning or thinking from developers. Every time I've allowed logic in config files, I've regretted it.
The main annoyance about JSON is lack of commenting ability. This is a point I agree with but not a big deal IMO, it's still better than all other alternatives.
JSON is ideal in terms of being both machine-readable and human-readable. Some people would argue that YAML is more human-readable, but it's definitely less machine-readable. With YAML, there are too many situations where some random code somewhere will remove all the new lines and tabs (for whatever reason) and mess everything up. JSON is resilient to machine sanitization. JSON is simple, robust and readable.
I usually use libconfig for configuration, which is better than json and yaml and has implementations in many languages (although I'm not aware of any javascript implementation).
Kotlin and kotlin script seems to work reasonably well for Gradle. Gradle unfortunately has a bit of groovy legacy, which makes it a bit convoluted for some things. But Kotlin DSLs are quite nice for configuring things and it removes a lot of ambiguity when your editor can tell you a property doesn't exist or that a list instead of a string is expected. or help you autocomplete things that are allowed.
Basically the ultimate in configuration languages is something that has a minimal syntax but provides enough structure for tooling to help you validate things, provide autocomplete and other help, etc. There are now several languages emerging that have both static typing and some level of support for internal DSLs. Ruby sort of pioneered a lot of configuration DSLs back in the day but since it was dynamically typed, tool support for these DSLs was a lot harder. With languages like Kotlin, Swift, C#, Rust, etc. you get enough expressiveness that you can do similar internal DSLs but with the advantage of more robust tooling support.
With gradle, there's three big phases: initialization, configuration, then execution. (See https://docs.gradle.org/current/userguide/build_lifecycle.ht...). I'd think you'd need to do something similar with any structured configuration DSL - have distinct phases where you may "init, plan, apply", etc.
This, it's a pretty complex problem to have. But that complexity may be worth it as part of a "developer productivity engineering" effort, where you're really looking to use data to minimize the time between a configuration change, and the time it's applied in a runtime environment. Doing that is a hard problem.
From that perspective Groovy is much nicer ...
Declarative languages are a work around for bad APIs.
Specifically, the project uses TypeORM and it was nice to be able to import the types for the DB connection options to make sure I had all of the necessary properties.
I was using ts-node to run server-side without compilation, which made importing an `app.config.ts` file quite elegant, IMO.
It's a nice balance, I think - most of the time you can treat it as a normal config file, but if you need to drop into some programmatic stuff you can. I've done this in my own dotfiles to add custom package definitions before contributing them upstream, for example: https://github.com/jfrederickson/dotfiles/blob/master/guix/g...
God, what are we being punished for?
- doesn't have comments
XML uses the same <!-- stuff here --> SGML-style comments that HTML does
- bits of configs can't be reused
XML can include other XML via XInclude: https://www.w3.org/TR/xinclude/
- can't contain any logic
You can technically put logic in XML using XSLT (https://www.w3.org/TR/2017/REC-xslt-30-20170608/), but I would ask if one SHOULD put logic into your configuration. I personally would recommend against it if at all possible. Keep configuration as declarative as possible and put the logic in your program. It's better to have a configuration logic module or use dependency injection than put logic in your config.
- Programming language constructs are reinvented from scratch
Again, XSLT could be used, but should your configuration really need to be Turing-complete? Granted, XML is complicated and not without criticisms, but it is mature, standardized, and well-supported. Python has great XML support: https://docs.python.org/3/library/xml.html
- can't be validated
XML can be validated against XML Schema (https://www.w3.org/XML/Schema), Relax NG (https://relaxng.org/), and possibly other standards.
- implicit conversions and portability issues
XML can be used portably between different platforms and programming languages depending on how the data is represented. XML Schema defines a way to describe data types that is flexible and interoperabile: https://www.w3.org/TR/xmlschema-2/
Again, I know it's not the popular answer, but I was surprised that it was not even mentioned in the article.
Moving up the enlightenment chain:
or its brace-and-quoteless friend: place a lot more emphasis on the payload and less on the packagingWith XML and other "fully qualified" syntaxes the named closing tag makes it easy to visually see where a block starts and stops. Humans parse the text too. This begs the question: Do you read config more or do you write config more?
Braces and parens all look the same, so its harder to visually match them. "But wait," you may say, "my editor / IDE does brace and paren matching so I can easily see and jump to the corresponding brace or paren." True, but doesn't your editor / IDE also support or have plugins for automatic HTML / XML end tag insertion?
Personally, I am equally comfortable with XML and JSON style configuration, but I find the brace-and-quoteless style unsettling because I believe that whitespace is a poor choice for delimiting structure. Your opinion may vary and that is fine. Do what works best for you.
Expressiveness and flexibility are good so that you can define configuration to meet the needs of the system or application you are building.
XML Schema, Relax NG, etc. can be used to specify how the XML configuration should be structured, limits on data types, required versus optional configuration items.
As I said before, XML gets a lot of flack for being verbose, ugly, and complicated, but it is mature, widely-supported, and might be worth considering depending on your needs.
This is well-formed xml and good way to confuse people:
A few of OPs suggestions are not Turing complete, but rather total programming languages as is the case with Dhall.
Well, definitely not XSLT.
Case in point, PHP's "Smarty" templating engine. Because lord forbid people learn to use PHP as a templating language (which is ALREADY IS), when instead they can learn to use an ad hoc, informally-specified, bug-ridden, slow implementation of half of PHP, which is implemented in PHP, and dynamically generates PHP code.
https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
>Greenspun's tenth rule of programming is an aphorism in computer programming and especially programming language circles that states:
>Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
Your templating engine sucks and everything you have ever written is spaghetti code (yes, you)
http://www.workingsoftware.com.au/page/Your_templating_engin...
>This is where folks will start to get technical about why their code isn't spaghetti code.
>Oh no! Not me because I use (Django|Rails|Haskell|Node.js|Symfony|CakePHP|Tornado|Plone) and the superior templating language (Django Template|ERB|Mustache|PHP|TAL|Smarty) doesn't allow business logic in templates and enforces a separation of concerns
>-You, the reader, December 2011
>The problem is that in every single one of these circumstances, the templating engine has 2 primary responsibilities:
>The fragmentification and inclusion of common interface assets (to avoid duplication and maintaining the same code in multiple files) - eg. headers, footers, sidebars and the like
>The output of dynamic content
>The output of dynamic content requires logic (if/else, loops being about the bare minimum).
>The creators of templating languages have, to varying degress, availed themselves of a false dichotomy, namely that there exists "presentation" logic and "business" logic, the former of which belongs in templates and the latter of which has absolutely no place in templates and must be banished immediately.
>I say false dichotomy because this is an arbitrary separation - a choice made by the person who created the templating engine and is completely subjective.
Yet another reason smarty sucks
https://www.sitepoint.com/community/t/yet-another-reason-sma...
>Which brings me right back to my chief criticism of smarty which to this day it’s adherents can’t answer – what the Hell is the point? [...]
>I hated smarty from the day I tried it and I was always wondering why would anyone use it since you’re forced to learn some weird syntax that’s so similar to php - so why not use php from the start?
https://news.ycombinator.com/item?id=4605887
>PHP is itself a template engine. Bolting a template engine on top of that makes no sense whatsoever.
https://news.ycombinator.com/item?id=20736493
>+1 for Turing complete programming languages instead of half-assed config lan...
There's examples of this working effectively: millions of people can use excel to perform computations on groups of cells - the formulae they're entering include basic programming constructs like assignment, selection, loops, etc - yet it's not presented as a "programming system", which otherwise seems to put people off.
I recall reading an anecdote about the TECO editor (or some other emacs predecessor). Where the editor was being used by secretaries and other non-programmers, but they'd have no problems configuring it with the documentation they had available - they were never told that they were actually programming when they were doing so.
Perhaps it's a bit much to give the user a full-blown programming language for their configuration and expect them to have no problems, but it seems like limited programming concepts can be learned by just about anyone if presented in the right way.
And as you said, presentation is key. Presented as "use these tools to set this up" things could go well. Presented as "write this in python" it may not go at all.