I love seeing useful languages that aren’t turing complete. Tooling gets much more interesting when you don’t have so many impossibility results around all the interesting stuff.
What practical advantages are there? Someone can always write a program that runs until the heat death of the universe even in a Turing-incomplete language.
Totality checking like that in Idris is one nice example. That's kind of circular though - it isn't Turing complete because it has totality checking (i.e. you must prove the program terminates).
In theory that's true for many of the kinds of Turing-incomplete languages we care about. (Eg it's not true for JPG or for (proper) regular expressions.)
From Dhall's docs (emphasis mine):
> Note that a “finite amount of time” can still be very long. For example, there are some short pathological programs that take longer than the heat death of the universe to evaluate. The main benefit of evaluation being finite is not to eliminate long-running programs but to make them significantly less probable. In practice, you will discover that you will rarely author a configuration file that takes a long time to evaluate by accident.
> For example, Dhall does not provide language support for recursion. If you try to define a recursive expression or function you will get a type error. Lists are the only recursive data structure and the only way to build or consume lists is through safe primitives guaranteed to terminate, like List/fold. This restrictive programming style keeps code simple and makes expensive code more obvious (both to the code author and reviewer).
I guess I don’t care much if a configuration language takes a long time in some pathological case (a user writes a loop that doesn’t terminate). Such cases are already too rare to justify a dedicated language. And if the concern is bad actors, then a bad actor wouldn’t have to try to hard to get a Dhall program to run for a long time (copy paste something from the Internet). The more compelling reason to use Dhall is the static type system which is sadly uncommon among configuration languages.
So common that it justifies a change of language? "Oops, this thing that should have run for 5s has run for 30s, something must be wrong, <cancel>". You can even automate that reasoning by way of a timeout. In a hypothetical world where I'm choosing between two otherwise exactly equal configuration languages, I suppose this would give the advantage to Dhall, but honestly I'd probably pick TypeScript because making an entire org learn a new syntax is far more trouble than that saved by non-Turing-completeness.
The major advantage of a language that isn't Turing complete is not having the major risk inherent to Turing complete languages: asking if any non-trivial program will produce any given result or behavior is undecidable[1].
> write a program that runs until the heat death of the universe even in a Turing-incomplete language.
The Halting Problem is just a simple example of program behavior. The undecidability extends to any other behavior. Asking if a given program will behave maliciously is still undecidable even if we only consider the set of programs that do halt in a reasonable amount of time.
When you are using a regular language or deterministic pushdown automata, questions about the behavior or even asking if two implementations are equivalent is decidable. It is at lest possible8 to create software/tools to help answer the question "is this input safe." When you use a non-deterministic pushdown automata or stronger, you problem becomes provably undecidable*,
I highly recommend the talk "The Science of Insecurity"[2].
> The major advantage of a language that isn't Turing complete is not having the major risk inherent to Turing complete languages: asking if any non-trivial program will produce any given result or behavior is undecidable[1].
It's not obvious to me that I should care about this property in a configuration language. For a given configuration use case, I probably have a good idea about the extreme upper-bound for a correct program--say, 5s. If the program runs for 30s, the supervisor kills it.
> The Halting Problem is just a simple example of program behavior. The undecidability extends to any other behavior. Asking if a given program will behave maliciously is still undecidable even if we only consider the set of programs that do halt in a reasonable amount of time.
What's a malicious action in a configuration use case that a Turing complete program could muster but not a non-Turing-complete program?
I absolutely loved using this tool to generate many instances of Datadog resources in terraform json. Not only did it simplify a nasty mess of dynamic typed and schema-less bash/python/mako, it taught me a lot of functional programming paradigms along the way.
This is a by-product of Dhall coming from the Haskell community where this formatting was preferred instead of final trailing commas. It has been internalised and was carried on. I say this as someone who looks at this code and thinks "This is totally fine." -- but I admit, I'm environmentally damaged.
Yes. And, alas, you can't just allow a final trailing comma everywhere in Haskell.
Eg ("Foo", 2,) is different from ("Foo", 2) in Haskell thanks to TupleSections. For innocent bystanders: in Haskell ("Foo", 2) is the tuple you'd expect it to be. But ("Foo", 2,) is a function that takes another argument and creates a three-tuple. A Python equivalent would be
How so? Leading comma puts the terminator on the line after what it's terminating (which is why it bothers me, can't speak for others). This doesn't do that.
Yeah, I also really dislike this mixing up of indentation levels because of a language limitation. The "{" denotes a container, and the "," a separation between items of said container, for me it makes sense for them to be nested a level deeper
That's one way to interpret things, but not the only way.
In practice, this style reads just fine once you get used to it. Indentation is mostly there as a human convenience, so just has to work well with human brains (and be understood by a computer, if it's significant), but doesn't have to necessarily follow some abstract unified theory of syntax trees.
both symbols are delimiters, one can choose to align them together even when they are not the same character. Opening and closing braces aren't the same character either, but people have been aligning them for ages, I don't see a reason why commas, while being part of the same expression, should not follow the same principle.
For me it's not about being the same character (like you've mentioned, opening and closing braces aren't either), it's about commas and braces indicating different things in the hierarchy. Not to mention the symmetry breaking: an opening brace together with data in a line, then a lonely one at the bottom.
In a series of declarations, the lonely closing brace at the bottom can be treated as a substitute for an empty line between two entries, as it produces a similar sparse spacing.
I much prefer this style, I use it wherever I can because I can comment out lines without a trailing comma causing a syntax error. SQL, Ruby… wherever I can.
Try it, I doubt you'll go back (unless someone's stupid parser doesn't let you).
What's the point of allowing a trailing comma? A delimiter symbol is a delimiter symbol, be it a an opening curly brace or a comma between elements, and one can simply align delimeters.
They should at least reveal the answer somewhere immediately close by. I couldn't spot the error after a few minutes, came to the comments to find out what it was. Not a great introduction because you're asking someone to spot an error in something they don't know the syntax or ruleset for.
This opinion is ironic because this is exactly what the author intended to describe -- the typo IS hard to find.
The "Hello, World" example is nothing more than JSON, showing a string repeated 3 times. On the third time, "bill" is misspelled with "blil". The next tab shows how Dhall uses a variable definition to prevent this type of error.
I too spent the whole time scrutinizing the syntax and ignoring the values, since this is an intro to the language and not to Bill’s dotfiles. I was ultimately unable to deduce the presumed syntax error and independently came to the same misconclusion.
I mean... from context, it's probably a mistake, but on the other hand we don't actually know the structure of that filesystem or the author's intentions.
Meh, I think you're pointing out why doing this with a variable is better. In other words that mistake is the entire point of the example. Perhaps the definitions tab could make it more clear what the mistake was.
Yeah, I spent 3 minutes looking for a syntax error in 5 lines of code. I couldn’t find anything. I’m obviously too dumb to figure out this language so I guess I’ll just stick with YAML.
Don't. Use helm, kustomize or a decent code language.
Dhall will constrict, slow you down, make onboarding a nightmare, and ultimately be as brittle as other alternatives (Only it's harder to find where it broke).
I cannot advocate against dhall enough.
this is what I thought too. I've enjoyed working with helm and totally recommend it but was wondering if I'd missed something. A templating language and not an actual programming language seems to be the right balance for config
Dhall is my favorite configuration language that I never get around to using.
I manage DNS in Terraform, and since every Terraform provider uses different objects definitions, and every object definition is rather verbose, Dhall would be a way to specify my own DRY types and leave the provider-specific details in one place. Adding new DNS entries and moving several domains between providers would be a matter of changing fewer lines.
tanka helped me make peace with k8s (yaml) or at least made for a good learning environment for k8s than the other options. Wonder about other peoples experience or their rite of config tool passage
We use https://github.com/octodns/octodns for some of our DNS records. It's flexible, much faster than Terraform for thousands of records, and the maintainer Ross has been responsive on issues and pull requests. Also see Cloudflare's blog for how they use it
I have been recently looking for a program that does a similar thing, ie simplifies creation of configs. And I did not find a suitable one by Googling. Then I realized the PHP is perfect for that and I already know it! The output of PHP does not have to be HTML or other web stuff, instead it can be a config file.
I too don't want to or would use a bash script as a config file. That's the point of dhall: you get good static types and it's not turing complete, so you get a lot of the safety and code reusability tools without the pitfalls of a general purpose programming language.
And, have you ever worked with giant configs in json or yaml? It becomes incredibly painful to manage.
I think more programming power is useful in the service of stopping bugs; for instance, you could have a language where the part that "does stuff" isn't Turing complete, but the part that "stops incorrect stuff from happening" (the type system) is.
My biggest pain when doing this is: set up an environment configuration. Then set up a test or stage environment that matches it. Then modify the environment over the next few years keeping the stage and production environment configs in sync so that testing and validation are useful (the stage configuration actually matches production so that success or failure in staging predicts the same in production) before going to production.
Even something as simple as variable substitution becomes very useful in these cases so that configuration can be a single file and substitution delivers the staging or production config. Functions and operators allow more complicated configurations to remain DRY. Checks prevent the stage servers from using the production database connection string, etc.
People tend to evaluate these things based on what cool and neat things they can do it with. Not whether telling people they need to learn Haskell before they can update what would've been a 30 line configuration file on the project they just joined is a good use of peoples' time.
Is it? My takeaway is "oh cool, first-class types". Experimenting with this, I can write the following:
let ConfigOf : Type -> Type = \(type : Type) ->
{- What happens if you add another field here? -}
{ home : type
, privateKey : type
, publicKey : type
}
let Config : Type = ConfigOf Text
and the rest of the example still works and evaluates the same.
Also in a later example it has the expression `generate 10 Config buildUser`, which also works because of first-class types. Instead of needing generics, you just take a type as a regular parameter.
Why not instead use one of the popular programming languages (C#, Java, ...), possibly with some dedicated library built on top of it? Also, reminds me of the Heptagon of Configuration [1].
I work on a deployment tooling team. In 2019/2020 we did a deep dive into Dhall vs. Jsonnet^1 for standardizing config and kubernetes templating across my company (Zendesk). We ended up going with Jsonnet (although some Dhall evangelists in the company have kept the dream alive!), which I think is a more approachable language for many, but Dhall has a lot of cool features and good things going for it.
Jsonnet is far from perfect, but it gets the job done pretty well and has been relatively easy for engineers across the company to pick up.
That said, after a 3+ year journey, the shortfalls in our original design have become more noticeable and we're giving more thought to writing a more robust tool using a more traditional language like Go to solve some of our configuration/deployment and data templating problems.
Cue^2 is something I'm keeping an eye on these days as well.
I like Dhall for what it gives you with type safer etc but the compile times are extremely slow on a large project and type signatures can get real long/hard. It is bit of a Dhall monolith though ;)
> we're giving more thought to writing a more robust tool using a more traditional language like Go to solve some of our configuration/deployment and data templating problems.
You might want to give Pulumi (no affiliation) a look. I've been using it with Typescript, but it supports Go, too.
Yeah, I really like what I've seen from CUE, and obviously the core team behind it is the real deal. I've never worked at Google but I've done quite a bit of research into Borg/BCL. AFAIK Jsonnet is basically a direct decedent of BCL, whereas CUE is the next evolution that tries to fix what BCL got wrong.
CUE was in its infancy when we were evaluating Jsonnet (and I wasn't even familiar with it at the time). If we were picking a data templating language today there's a good chance we would choose it. We very well may end up migrating to it or incorporating it in some way.
Without knowing anything about Cue, just be careful with choosing to a technology only because it scales to the largest infra on earth, it doesn’t mean it scales equally good for smaller deployments.
> In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M.
It's... not obvious to me that that's a good thing? The ratio of configuration code to code in the things being configured being that close makes me think that BCL is something that's ill-suited to what it's now being asked to do but there's too much of it to realistically replace.
Maybe it's amazing and the problem it's solving is so complicated that even a language designed very well for solving that problem leaves you needing a lot of it, but I don't really consider "a staggeringly large amount of code has been written in this language" to necessarily be an endorsement of that language's quality. Citing Google's 300M lines of Java would also be a silly reason to pick Java in a project where you'll never interact with the Google ecosystem.
The number of lines of BCL at Google says little-to-nothing about the efficacy of the language itself, it's more of a reflection of the complexity and scale of the _systems_ its used to configure.
I have personally written several thousand lines GCL (the generic version of BCL used at Google) and I can say that it can be pretty frustrating.
The difficulty and complexity of defining configurations using it really depends on the system you are configuring since you are (generally) just defining a set of static fields that are packaged into a protobuf and fed into whatever system you are working with.
Outside of syntax issues, it's up to the system you are configuring to provide concise config semantics and helpful error messages
BCL is great, it's just lived out by its longevity.
But the point is that this large scale application offered the space for exploring the space of configuration as code, infra as code, and many relevant technical problem space for designing and deploying configuration.
Same here. It did some cool stuff but really bent your mind.
Agree with the other comment that the amount of it at Goog should have a mixed interpretation. Obviously it was useful, but although Google systems were complicated, it still doesn't seem right that it's on the same magnitude as the main system langs.
It was in fact the running joke that everyone knew everyone hated BCL.
But the issues with it were well known, so it makes sense to me that after learning all that the guy who invented it could have come up with a really good successor.
You are hating BCL because it's not loved by anyone. It's like the poor kid who were supporting the prodigy that is the Borg, but is actually supporting the Borg ecosystem on its back, where the prodigy kid just grab all of the acclaim.
That's just typical of almost all human involved affairs. Some dudes hard support but get 20% of what they deserve. Whereas some few get 800%.
As for BCL, it literally had no investment since 2010, yet still reliably March on ward with little support, in which case 99.99999% of software would simply die into oblivion. BCL just shows it's power and strength.
To name the importance of any single software for Google's success, the 120k lines of CPP code in borgcfg stands on a high peak that look down upon all the other dwarfs with ease.
But I sure can tell you a few easy changes for BCL ecosystem can make a huge difference:
* Testing: despite the common knowledge of testing everything, BCL has not testing facility. No man power to support this easy (easy in relative scale) feature.
* Code search: BCL has inheritance like semantic, but no support in code search to navigate the code base. That's ridiculous. If anyone claim Google cpp code can be supported by lack of code searching navigation, then BCL is rightly to be called mysterious and lack sane language design.
* Packaging: BCL is always halg mixed and half separated with executable. That's just plainly stupidity.
* Lack of sane abstraction from Borg: Borg was designed without package concept. Borg's package was entirely an invention by BCL and borgcfg, as an idempotent shell command executed before starting Borg job. Go figure how much a shaky ground BCL is relying on. And how much blame Bcl has to suffer on behalf of Borg's lazy design.
It certainly has that feel... that you can make statements (express constraints) about a given configuration item — in any order — and the final result falls out (having satisfied all constraints or "errors").
Whilst I like the look of Cue... this argument reminds me of the Kubernetes trap
( We should use it... because Google does it and they're huuuuuuuge don't you know etc )
This is actually one of my main concerns with CUE and something I'd like the community overall to keep in mind. There is a high chance in my opinion that CUE ends up being adopted as a silver bullet for configuration just for being cargo culted as something that is from Google and the drive for dependence by open source users sometimes, sort of similar to what happened with Kubernetes. That modules and package management in CUE are modeled after what Go did is also questionable IMO but makes sense since the project is Go based.
CUE is a good tool for validation and has its uses, but Dhall has some really neat innovations that make it an exciting project so worth looking at both at least and compare first.
ytt lets you embed logic via a python-subset (starlark) and also provides "overlays" as a "replace/insert" mechanism. and all valid ytt files are valid yaml files, so they can be passed-through other yaml parsing stages.
Thanks for sharing! We've done a little experimenting with ytt (we already use several tools from Carvel/k14s, mainly vendir and kbld), but it's been a while...probably a year or more...since I've played with it. Need to give it another look.
Just my 2cents - ehen I started at current employer, we had a huge, convulted dhall project for kube. We ended up switching to a real language (python in our case due to reasons, Go is a more correct choice) and are very pleased with the results.
I thought the same thing. As a Node.js dev, I can quickly create a .mjs file and use a `module.export` to return a JavaScript object who contains the configuration. Thus, i can use template string and/or function to do what I want. I can even load JSON files natively
Unless you count the ability to do imports from around the web. But you can still make it side-effect-less by using caching and semantinc hashes on those imports?
these are side-effects performed by the compiler's runtime in a sandboxed environment. Dhall scripts are unable to perform side-effects on their own. This is a completely different setting compared to unrestricted side-effects available within a script written in general-purpose language.
jsonnet seemed like a great idea to me, but I've experienced extremely low performance. My Kubernetes manifest with a couple hundred lines of code in 4 files, which rendered instantly in Python before and renders instantly now with Helm, would take 10-20s with jsonnet. The "lazy evaluation" would probably go into some quadratic or exponential behavior, evaluating the same thing many times, but I sure couldn't see why (it was very straightforward code) or where (no debug tools...).
I really wouldn't use it for anything until tooling improves (but then against I'd much rather use something like Starlark).
> jsonnet seemed like a great idea to me, but I've experienced extremely low performance.
Although each implementation of jsonnet has some quirks, take a look at sjsonnet^1 (scala-based) or go-jsonnet^2 for improved performance. We generally prefer go-jsonnet.
There's also a Rust version^3 that claims to be the fastest yet^4, but I haven't experimented with it at all.
I wonder if a version of Python with types with some way to force programs to be total would be a 'good enough' substitute. JavaScript has `"use strict";`, `# use total` could ban recursion and loops over non-constants. You'd have to work hard to ensure things were really total, e.g. ban assigning functions to variables, but maybe you could make it hard enough to pay off.
Starlark is a version of this, and it's used by Bazel in particular. It's very cool tech, and sidesteps the difficulties of ensuring Python works without shipping a specific Python interpreter.
I believe you may want a non-turing complete language for strict configuration.
But sometimes, as you indicate, you want VERY dynamic configuration. But I would argue then that such logic goes in your application itself and is not in fact part of your configuration.
Most of the time with configuration files, you want to be able to do at least basic validation on the config without having to run the whole piece of software. Python has no native typing, so there's nothing to stop the user from creating a configuration file where, for example, they have set a field that is intended to specify a port number, as a string. This error will become obvious when you run the software (and it presumably fails some way in) but for some use cases this is too late. With dhall you can specify that the field for the port number needs to not just be an integer, but be a natural number, so you're able to catch plenty of config errors nice and early, and can go some way to validating the config in isolation from the software it's intended to be used for.
I said this above as well: ytt (https://carvel.dev/ytt/) lets you embed starlark into valid yaml, among other cute tricks for managing biz-logic in configs.
Huh. Clever strategy, though I'm not sure it'd be something I'd use...
It is a pretty decent "take your existing yaml and make incremental improvements" setup though. Rewriting all of your config from scratch is rarely an enjoyable experience.
Sometimes you just explicitly want to output text or a structured format, but you'd rather have a template language do it because it being a lot less powerful actually makes it easier to write close to the output format.
I've always wanted an excuse to use this but never had a reason. The closest I come is using Nix, and I know there is a Dhall-to-Nix compiler, but Dhall can't represent everything possible in Nix I haven't seen any good reason to use it.
I really hope that one day Google can open source GCL, which so many languages have been inspired by without ever really being as good as it is. They all claim to address pitfalls in GCL but at the end of the day they're just worse than GCL.
And yet, even if they did it right now, it'll be over there in the pile of "yet another configuration language" for the same reason we already have like 50 of them
Can you say more about what GCL does better than all of the open source ones?
Anecdotally, I've heard a lot of GCL horror stories, and many Xooglers have chosen to create things like Jsonnet or Skycfg (https://github.com/stripe/skycfg) instead.
It creates nested data without looking like the end format of the data (why I really can't get into jsonnet), it's just obviously its own language but is pretty minimal.
Despite the minimalism it is not necessarily simple. There are features like inheritance and late binding. They can be quite complicated, but the thing that really stands out about GCL is it's not complicated to use. The simple cases are easy, straightforward, and readable. The complicated cases are made possible.
I've used it extensively to store and transform data that has to be manually edited by humans. It's a really great format because I can define all the translation rules which can get pretty complicated, but the complexity I expose to the humans who are writing the data is minimal. But if they need to do some transformations on the data or even just, like, string substitutions...the language is there and they can use it. That's what makes writing your data in a configuration language nice.
(Honestly though I think jsonnet is at least vastly superior to skycfg and the whole starlark ecosystem. I swear I'm so sick of languages that intentionally look like Python without being actual Python!)
> However, the early design of GCL went for something simpler that coincidentally was also incompatible with the notion of graph unification. This simpler approach proved insufficient, but it was already too late to move to the earlier foreseen approach. Instead, an inheritance-based override model was adopted. Its complexity made the earlier foreseen tooling intractable and they never materialized. The same holds for the GCL offsprings that copied its model.
Needless to say I disagree with the Cue author. I think the inheritance-based override model is fantastic and has made for a great and straightforward configuration language.
I think Dhall is good at pushing forward some ideas, but honestly I feel like Skylark (Python but not turing complete) just feels like the right way forward. Being able to specify dynamic functionality in a configuration file, when paired with a good configuration API, really makes stuff straightforward IMO. Gunicorn is the best example of this.
We have all of these tools that try to propose declarative configuration, then run into the fact that people really do want dynamic systems with some abstraction capabilities, and then have to overlay that into their systems.
I feel increasing alone in this position but I absolutely hate those Python-like languages.
They look like Python, and they occasionally act like Python. But occasionally they don't, and the mask of looking like Python obscures those times. Then you never really know which Python facilities you have and which ones you don't.
191 comments
[ 3.2 ms ] story [ 222 ms ] threadFrom Dhall's docs (emphasis mine):
> Note that a “finite amount of time” can still be very long. For example, there are some short pathological programs that take longer than the heat death of the universe to evaluate. The main benefit of evaluation being finite is not to eliminate long-running programs but to make them significantly less probable. In practice, you will discover that you will rarely author a configuration file that takes a long time to evaluate by accident.
> For example, Dhall does not provide language support for recursion. If you try to define a recursive expression or function you will get a type error. Lists are the only recursive data structure and the only way to build or consume lists is through safe primitives guaranteed to terminate, like List/fold. This restrictive programming style keeps code simple and makes expensive code more obvious (both to the code author and reviewer).
(Same for the recursive equivalent.)
The major advantage of a language that isn't Turing complete is not having the major risk inherent to Turing complete languages: asking if any non-trivial program will produce any given result or behavior is undecidable[1].
> write a program that runs until the heat death of the universe even in a Turing-incomplete language.
The Halting Problem is just a simple example of program behavior. The undecidability extends to any other behavior. Asking if a given program will behave maliciously is still undecidable even if we only consider the set of programs that do halt in a reasonable amount of time.
When you are using a regular language or deterministic pushdown automata, questions about the behavior or even asking if two implementations are equivalent is decidable. It is at lest possible8 to create software/tools to help answer the question "is this input safe." When you use a non-deterministic pushdown automata or stronger, you problem becomes provably undecidable*,
I highly recommend the talk "The Science of Insecurity"[2].
[1] https://en.wikipedia.org/wiki/Rice%27s_theorem
[2] video: https://archive.org/details/The_Science_of_Insecurity_ slides: [pdf] https://langsec.org/insecurity-theory-28c3.pdf
It's not obvious to me that I should care about this property in a configuration language. For a given configuration use case, I probably have a good idea about the extreme upper-bound for a correct program--say, 5s. If the program runs for 30s, the supervisor kills it.
> The Halting Problem is just a simple example of program behavior. The undecidability extends to any other behavior. Asking if a given program will behave maliciously is still undecidable even if we only consider the set of programs that do halt in a reasonable amount of time.
What's a malicious action in a configuration use case that a Turing complete program could muster but not a non-Turing-complete program?
Looks great otherwise
Eg ("Foo", 2,) is different from ("Foo", 2) in Haskell thanks to TupleSections. For innocent bystanders: in Haskell ("Foo", 2) is the tuple you'd expect it to be. But ("Foo", 2,) is a function that takes another argument and creates a three-tuple. A Python equivalent would be
lambda x: ("Foo", 2, x)
This actually works just fine.
In practice, this style reads just fine once you get used to it. Indentation is mostly there as a human convenience, so just has to work well with human brains (and be understood by a computer, if it's significant), but doesn't have to necessarily follow some abstract unified theory of syntax trees.
Try it, I doubt you'll go back (unless someone's stupid parser doesn't let you).
(And that's what the comment you reply to suggests: make Dhall allow a final comma.)
Another ~similar project not mentioned in that thread is https://ucg.marzhillstudios.com
Nope, so now I have no incentive to use your config format because it's established something is wrong and it's completely non-obvious.
Thanks for not wasting my time, I guess.
The project looks cool though.
The "Hello, World" example is nothing more than JSON, showing a string repeated 3 times. On the third time, "bill" is misspelled with "blil". The next tab shows how Dhall uses a variable definition to prevent this type of error.
On small viewports, the “right-hand side” is down page and off screen, and the instructions I’m following reference only tabs visible above.
Would be cool if they had a three panel story:
* using json with the typo
* using dhall with the typo
* using dhall with variables to avoid the typo
Might be easier to understand.
I manage DNS in Terraform, and since every Terraform provider uses different objects definitions, and every object definition is rather verbose, Dhall would be a way to specify my own DRY types and leave the provider-specific details in one place. Adding new DNS entries and moving several domains between providers would be a matter of changing fewer lines.
Dhall also has Kubernetes bindings:
https://github.com/dhall-lang/dhall-kubernetes
Although I'm tempted to just stick to Helm here: even though it's less type-safe, Dhall's verbosity makes me reconsider.
I'd like to hear if anyone has used dhall-kubernetes if they like it.
https://tanka.dev/
Are there any pitfalls you have learned to avoid?
LE: The "garbage collection" feature is very interesting, but I didn't get to experiment with it enough yet: https://tanka.dev/garbage-collection
"Comparisons between CUE, Jsonnet, Dhall, OPA, etc." https://github.com/cue-lang/cue/discussions/669
I would not want to use a bash script as a config file, for example.
And, have you ever worked with giant configs in json or yaml? It becomes incredibly painful to manage.
Even something as simple as variable substitution becomes very useful in these cases so that configuration can be a single file and substitution delivers the staging or production config. Functions and operators allow more complicated configurations to remain DRY. Checks prevent the stage servers from using the production database connection string, etc.
Clear as mud.
[EDIT] I mean, you can almost "who's on first?" this.
"OK, what type do you want this to be?"
"Type."
"Yes, what type do you want it to be?"
"Type."
"Great, ok, yes, the type, what type is Config?"
"Type."
"WHAT IS THE NAME OF THE TYPE YOU WANT CONFIG TO BE???"
"Type."
flips desk
Also in a later example it has the expression `generate 10 Config buildUser`, which also works because of first-class types. Instead of needing generics, you just take a type as a regular parameter.
[1] https://matt-rickard.com/heptagon-of-configuration
Jsonnet is far from perfect, but it gets the job done pretty well and has been relatively easy for engineers across the company to pick up.
That said, after a 3+ year journey, the shortfalls in our original design have become more noticeable and we're giving more thought to writing a more robust tool using a more traditional language like Go to solve some of our configuration/deployment and data templating problems.
Cue^2 is something I'm keeping an eye on these days as well.
[1]: https://jsonnet.org/
[2]: https://cuelang.org/
Have you looked at https://github.com/kris-nova/naml ? :)
You might want to give Pulumi (no affiliation) a look. I've been using it with Typescript, but it supports Go, too.
BCL code is the 3rd largest human written code in Google internal code base. In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M.
BCL configuration's large scale use probably is beyond any other infra as code use cases known to human.
And the learning and ideas over more than a decade, is manifested in CUE.
Personally, this is enough to convince me to comfortably ignore anything else on the market.
CUE was in its infancy when we were evaluating Jsonnet (and I wasn't even familiar with it at the time). If we were picking a data templating language today there's a good chance we would choose it. We very well may end up migrating to it or incorporating it in some way.
It's... not obvious to me that that's a good thing? The ratio of configuration code to code in the things being configured being that close makes me think that BCL is something that's ill-suited to what it's now being asked to do but there's too much of it to realistically replace.
Maybe it's amazing and the problem it's solving is so complicated that even a language designed very well for solving that problem leaves you needing a lot of it, but I don't really consider "a staggeringly large amount of code has been written in this language" to necessarily be an endorsement of that language's quality. Citing Google's 300M lines of Java would also be a silly reason to pick Java in a project where you'll never interact with the Google ecosystem.
Can it be used to successfully build and maintain configuration at extremely large volume and complexity scales? Yes.
Also we're not talking about using this language, but its spiritual successor.
I have personally written several thousand lines GCL (the generic version of BCL used at Google) and I can say that it can be pretty frustrating.
The difficulty and complexity of defining configurations using it really depends on the system you are configuring since you are (generally) just defining a set of static fields that are packaged into a protobuf and fed into whatever system you are working with.
Outside of syntax issues, it's up to the system you are configuring to provide concise config semantics and helpful error messages
But the point is that this large scale application offered the space for exploring the space of configuration as code, infra as code, and many relevant technical problem space for designing and deploying configuration.
Agree with the other comment that the amount of it at Goog should have a mixed interpretation. Obviously it was useful, but although Google systems were complicated, it still doesn't seem right that it's on the same magnitude as the main system langs.
But the issues with it were well known, so it makes sense to me that after learning all that the guy who invented it could have come up with a really good successor.
That's just typical of almost all human involved affairs. Some dudes hard support but get 20% of what they deserve. Whereas some few get 800%.
As for BCL, it literally had no investment since 2010, yet still reliably March on ward with little support, in which case 99.99999% of software would simply die into oblivion. BCL just shows it's power and strength.
To name the importance of any single software for Google's success, the 120k lines of CPP code in borgcfg stands on a high peak that look down upon all the other dwarfs with ease.
But I sure can tell you a few easy changes for BCL ecosystem can make a huge difference:
* Testing: despite the common knowledge of testing everything, BCL has not testing facility. No man power to support this easy (easy in relative scale) feature.
* Code search: BCL has inheritance like semantic, but no support in code search to navigate the code base. That's ridiculous. If anyone claim Google cpp code can be supported by lack of code searching navigation, then BCL is rightly to be called mysterious and lack sane language design.
* Packaging: BCL is always halg mixed and half separated with executable. That's just plainly stupidity.
* Lack of sane abstraction from Borg: Borg was designed without package concept. Borg's package was entirely an invention by BCL and borgcfg, as an idempotent shell command executed before starting Borg job. Go figure how much a shaky ground BCL is relying on. And how much blame Bcl has to suffer on behalf of Borg's lazy design.
ytt lets you embed logic via a python-subset (starlark) and also provides "overlays" as a "replace/insert" mechanism. and all valid ytt files are valid yaml files, so they can be passed-through other yaml parsing stages.
I heard you liked configuration languages, so I wrote a configuration language for your configuration language.
How long before you stick a Yaml config into your Go configuration library for easier maintenance? And so the cycle continues.
I really wouldn't use it for anything until tooling improves (but then against I'd much rather use something like Starlark).
Although each implementation of jsonnet has some quirks, take a look at sjsonnet^1 (scala-based) or go-jsonnet^2 for improved performance. We generally prefer go-jsonnet.
There's also a Rust version^3 that claims to be the fastest yet^4, but I haven't experimented with it at all.
[1]: https://github.com/databricks/sjsonnet
[2]: https://github.com/google/go-jsonnet/
[3]: https://github.com/CertainLach/jrsonnet
[4]: https://gist.github.com/CertainLach/5770d7ad4836066f8e0bd91e...
But yeah, you'd need to carefully select your subset of the language that is both useful and total, and practical to implement.
But sometimes, as you indicate, you want VERY dynamic configuration. But I would argue then that such logic goes in your application itself and is not in fact part of your configuration.
They have a great writeup on their safety here: https://docs.dhall-lang.org/discussions/Safety-guarantees.ht...
It is a pretty decent "take your existing yaml and make incremental improvements" setup though. Rewriting all of your config from scratch is rarely an enjoyable experience.
https://github.com/oilshell/oil/wiki/Survey-of-Config-Langua...
Dhall: A Non-Repetitive Alternative to YAML - https://news.ycombinator.com/item?id=20355405 - July 2019 (177 comments)
At the top of the main page.
See also the Hello World and other examples.
Damn network effect, ruining everything :-/
Anecdotally, I've heard a lot of GCL horror stories, and many Xooglers have chosen to create things like Jsonnet or Skycfg (https://github.com/stripe/skycfg) instead.
Despite the minimalism it is not necessarily simple. There are features like inheritance and late binding. They can be quite complicated, but the thing that really stands out about GCL is it's not complicated to use. The simple cases are easy, straightforward, and readable. The complicated cases are made possible.
I've used it extensively to store and transform data that has to be manually edited by humans. It's a really great format because I can define all the translation rules which can get pretty complicated, but the complexity I expose to the humans who are writing the data is minimal. But if they need to do some transformations on the data or even just, like, string substitutions...the language is there and they can use it. That's what makes writing your data in a configuration language nice.
(Honestly though I think jsonnet is at least vastly superior to skycfg and the whole starlark ecosystem. I swear I'm so sick of languages that intentionally look like Python without being actual Python!)
https://cuelang.org/docs/about/#history
From that link:
> However, the early design of GCL went for something simpler that coincidentally was also incompatible with the notion of graph unification. This simpler approach proved insufficient, but it was already too late to move to the earlier foreseen approach. Instead, an inheritance-based override model was adopted. Its complexity made the earlier foreseen tooling intractable and they never materialized. The same holds for the GCL offsprings that copied its model.
Needless to say I disagree with the Cue author. I think the inheritance-based override model is fantastic and has made for a great and straightforward configuration language.
We have all of these tools that try to propose declarative configuration, then run into the fact that people really do want dynamic systems with some abstraction capabilities, and then have to overlay that into their systems.
They look like Python, and they occasionally act like Python. But occasionally they don't, and the mask of looking like Python obscures those times. Then you never really know which Python facilities you have and which ones you don't.