People like to write entire plugin systems in their config languages. This tends to be a bad idea in my experience, but there's a steady demand for it.
One of the motivating cases is working around the Nix language's shortcomings:
> The Nix package manager: Nix is a declarative package manager using its own language for specifying packages. Nickel is inspired in part by the Nix language, while trying to overcome some of its limitations. It could be used instead of the Nix language.
Guix sidesteps this all by just using a proper language with macros for its configuration.
This is essentially what starlark does, although a slightly limited subset of python that's less prone to complicated things (no classes, so no multiple inheritance and no need to do what cue does).
As soon as you add some kind of macro system or any other content-rewriting, template-with-parameters or functional-abstraction mechanism, you may as well go full Turing. Otherwise people will try to simulate it using the available mechanisms, as they have for example with the C preprocessor, resulting in higher complexity than if you’d just given them Turing completeness in the first place.
So the only real choices seem to be either a pure data format or else full Turing completeness. There doesn’t seem to be a useful middle ground, unfortunately.
I think Dhall is an interesting point in this design space: powerful enough to write real functions, not Turing complete so all Dhall programs can be reduced to a normal form and compared for equivalence.
It seems to me the equivalence comparison will always have its limits. The Dhall documentation gives the example of `even(n + n)`, which Dhall is unable to recognize as equivalent to `true`. The feature is of course still useful, but I‘m doubtful the non-Turing completeness is absolutely essential to it, given its limitations.
In a total language, you can prove that `even(n+n) = true`. Dhall probably doesn’t bother with this because it’s focused on proving that your resulting data structures are equivalent: so, since functions are guaranteed to run in a bounded amount of time, it just runs all the functions it finds and then normalizes the resulting data structure. Equality between these data structures implies program equivalence.
I had to use Terraform at work to configure some SDN controllers . I never felt the need for an external language.
You can list a directory, read all the json that dir (in a single list/map comprehension/building expression) and use that to feed your modules that manage the actual terraform resources.
When you realize that terraform modules are functions, terraform become a usable functional language.
Why would I want to generate some terraform when I can simply use it to generate the desired state using my higher level data ?
Nickel does not generate Terraform. It is simply a language. The idea is that projects like Terraform could use it rather than create their own language out of a mess of JSON files.
In that sentence, I believe their meaning is that Terraform's language is already capable of generating configuration, and in fact this is a feature which is quite essential to Terraform's use. They acknowledge that all these projects (Nix, Terraform, Bazel, etc) already utilize relatively sophisticated programming languages for configuration. Nickel merely aims to be a better designed language for the purpose.
in general when you don't have it you end up with a mess of nested if statements. Or end up calling some outside script to inject some form of a function. Plus even without turing completeness you can still end up with exponential complexity.
It would be interesting to see some examples of actual configuration. At present, the examples seem to be limited to type definitions, nested let bindings and things like that.
Is configuration really so complex nowadays that we need entire languages (DSLs, really) for managing them?
Isn't there a better way? I feel like its so 'popular' to provide config hooks into libraries and systems, rather than exposing a Programmatic API that's at least unit testable to understand the results you get back, which makes it infinitely more reliable, IMO.
APIs are harder, I realize, because it forces you to really think about what you're going to expose and how before hand, rather than perhaps after the fact.
I also realize the tradeoff is that configs are suppose to be more limited by design (you can only express so much logic even when using some of these complex templating systems, they all reach some finite point of complexity). They're also suppose to be more portable (configs don't require runtimes per say, though that looks like its about to change!)
Arguably however, its gotten so complex to manage configurations that I would rather just have exposed APIs for systems so I can have better, robust testing in a language.
I realize this means some systems will no longer be language independent, but when I can use an API instead, I almost always do. Configuration management has just gotten out of control.
To think, people say webpack is 'config hell'. Ever try dealing with a complex templated Kubernetes config? Saltstack YAML Jinja2 madness?
I'll take webpack any day. At least its typed!
/rant
With that said, this looks neat. I can also stand by the idea that if there is a sane way to create a DSL like this that exports to any reasonable format (json, ini, properties file, YAML) then that actually, I will completely concede, may be worth it.
You can either have a configuration bit which is like "add_two_to_every_result: true", or you can have "result_post_processing = function(x) {return x + 2;}".
When working on complex systems, I know which thing is going to be easier to use for my "bespoke" systems, and which one is going to devolve into spaghetti messes. Unfortunately I think loads of system have gone for the former.
I think that stuff like grunt ended up being a mess because it tried to straddle the two, and webpack has made stuff cleaner in that respect, by accepting computation.
Almost every build system in the world, however, really doesn't like this model. The first CI system whose configuration file is, like, Lua, will become a billion dollar business overnight.
Special frustration goes to Bazel, which gives itself a cliff-shaped learning curve out of its "we really don't want to run arbitrary code" feeling, while actually having all the moving parts. You just need to dig through a bunch of "really really please don't do this" stuff to find the real docs.
> You can either have a configuration bit which is like "add_two_to_every_result: true", or you can have "result_post_processing = function(x) {return x + 2;}".
In this case there's an intermediate "increment_result_by: 2" that would be better than either. Once in a while there are use cases that are so undefined that "do whatever you want" makes sense, but usually this is an excuse to not think too much about what the user wants to do, and why.
The benefit of both of the limited solutions over the other is that it's possible to fully test the entire space of configurations. it definitely is possible to let this style of config devolve into spaghetti, but at least there's some hope of untangling it in the future, which is much less possible if you accepted essentially any result transformation some enterprising user had spent a week programming.
If the right solution for configuring your system is a full programming language, consider distributing your system as a library or framework, rather than a system that needs "configuration". If your library is fully tested and someone screws up in their own code, that's a different kind of problem than if someone provides a syntactically correct config that causes your system to fail. Blurring the lines between these kinds of problems is not going to make supporting your (non-programmer) users easier, in most cases.
Jenkins has persistent problems in my experience with reliability. We only ever really got shell scripts to work well with Jenkins at a previous place of employment years ago. The API layer was hard to leverage properly (part of what I was getting at in my rant. Which is why I get why configuration has taken over a lot of these kinds of complex systems)
I think a lot of this got fixed with Jenkins X because its underlying tech is just docker images. There again though you're mostly managing complex configuration rather than dealing with an exposed API layer
I feel like Lua solved this problem a long time ago and did it with a language fast enough for game logic and easy enough for children to learn. Although the s-expr people would justifiably argue scheme/lisp solved it much earlier.
Tcl is another contender for a configuration language that has been around for ages and ages. Although it's not as slick as Lua, which was of course originally designed as an extensible configuration language.
It’s nice to see a project that explains how it fits in the design space. Still after looking at this I get the feeling they haven’t really tried Cue in anger. I also think cue should be easier to introduce to a team precisely because it is restricted.
37 comments
[ 3.1 ms ] story [ 35.4 ms ] threadNickel: Better Configuration for Less - https://news.ycombinator.com/item?id=24858456 - Oct 2020 (60 comments)
One of the motivating cases is working around the Nix language's shortcomings:
> The Nix package manager: Nix is a declarative package manager using its own language for specifying packages. Nickel is inspired in part by the Nix language, while trying to overcome some of its limitations. It could be used instead of the Nix language.
Guix sidesteps this all by just using a proper language with macros for its configuration.
- What version of Python? They seem to have abandoned semver and are breaking changes across releases.
Granted, a Lua-like language would work well for some of these situations. Lua does have the "what version" problem.
Btw, there used to be a sandbox built in into stock python (Bastion), but they abandoned it (IRC because of security issues)
So the only real choices seem to be either a pure data format or else full Turing completeness. There doesn’t seem to be a useful middle ground, unfortunately.
You can list a directory, read all the json that dir (in a single list/map comprehension/building expression) and use that to feed your modules that manage the actual terraform resources.
When you realize that terraform modules are functions, terraform become a usable functional language.
Why would I want to generate some terraform when I can simply use it to generate the desired state using my higher level data ?
> terraform modules are functions
You seem to have found this feature useful. Doesn't that answer your question about why you would want a configuration language to have functions?
In that sentence, I believe their meaning is that Terraform's language is already capable of generating configuration, and in fact this is a feature which is quite essential to Terraform's use. They acknowledge that all these projects (Nix, Terraform, Bazel, etc) already utilize relatively sophisticated programming languages for configuration. Nickel merely aims to be a better designed language for the purpose.
https://github.com/tweag/nickel/tree/master/src/examples
Isn't there a better way? I feel like its so 'popular' to provide config hooks into libraries and systems, rather than exposing a Programmatic API that's at least unit testable to understand the results you get back, which makes it infinitely more reliable, IMO.
APIs are harder, I realize, because it forces you to really think about what you're going to expose and how before hand, rather than perhaps after the fact.
I also realize the tradeoff is that configs are suppose to be more limited by design (you can only express so much logic even when using some of these complex templating systems, they all reach some finite point of complexity). They're also suppose to be more portable (configs don't require runtimes per say, though that looks like its about to change!)
Arguably however, its gotten so complex to manage configurations that I would rather just have exposed APIs for systems so I can have better, robust testing in a language.
I realize this means some systems will no longer be language independent, but when I can use an API instead, I almost always do. Configuration management has just gotten out of control.
To think, people say webpack is 'config hell'. Ever try dealing with a complex templated Kubernetes config? Saltstack YAML Jinja2 madness?
I'll take webpack any day. At least its typed!
/rant
With that said, this looks neat. I can also stand by the idea that if there is a sane way to create a DSL like this that exports to any reasonable format (json, ini, properties file, YAML) then that actually, I will completely concede, may be worth it.
When working on complex systems, I know which thing is going to be easier to use for my "bespoke" systems, and which one is going to devolve into spaghetti messes. Unfortunately I think loads of system have gone for the former.
I think that stuff like grunt ended up being a mess because it tried to straddle the two, and webpack has made stuff cleaner in that respect, by accepting computation.
Almost every build system in the world, however, really doesn't like this model. The first CI system whose configuration file is, like, Lua, will become a billion dollar business overnight.
Special frustration goes to Bazel, which gives itself a cliff-shaped learning curve out of its "we really don't want to run arbitrary code" feeling, while actually having all the moving parts. You just need to dig through a bunch of "really really please don't do this" stuff to find the real docs.
In this case there's an intermediate "increment_result_by: 2" that would be better than either. Once in a while there are use cases that are so undefined that "do whatever you want" makes sense, but usually this is an excuse to not think too much about what the user wants to do, and why.
The benefit of both of the limited solutions over the other is that it's possible to fully test the entire space of configurations. it definitely is possible to let this style of config devolve into spaghetti, but at least there's some hope of untangling it in the future, which is much less possible if you accepted essentially any result transformation some enterprising user had spent a week programming.
If the right solution for configuring your system is a full programming language, consider distributing your system as a library or framework, rather than a system that needs "configuration". If your library is fully tested and someone screws up in their own code, that's a different kind of problem than if someone provides a syntactically correct config that causes your system to fail. Blurring the lines between these kinds of problems is not going to make supporting your (non-programmer) users easier, in most cases.
Jenkinsfile is written in Groovy and is not a billion dollar business that I'm aware of. Were you trying to emphasize the lua part?
Space from JetBrains has its build config written in Kotlin, for a more strongly typed version:
https://www.jetbrains.com/help/space/automation-getting-star...
I believe there are others but those are the two I have handy
I think a lot of this got fixed with Jenkins X because its underlying tech is just docker images. There again though you're mostly managing complex configuration rather than dealing with an exposed API layer
http://nickle.org/