why? It can't be because then if something goes wrong you need to debug your configuration, because you need to do that often enough with a non-programming language format for even less power offered.
If your config is so complex that it needs "debugging" then it's too complex.
The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability.
If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum.
And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
Why do you think so? I couldn’t disagree more. Task caching/avoidance, rich build domain model (source sets, configurations, tasks, extensions), huge plugin ecosystem.
My opinion is based on my experience integrating artifact generators (both code and config) into a build. I recall being funneled into building a plugin by documentation to deal with some idiosyncrasies of the generator. Whereas I just wanted to write a small script. In general I felt like there were a lot of extra hoops and concepts to learn to do basic easy shit. And there really isn't any payoff. For example I recall scripts and plugins have different capabilities. And that some things can't be expressed in untyped Gradle.
I also found that the user documentation was just a skosh above useless. It felt like it was more concerned with not being wrong than being helpful. Reminiscent of older javadocs in the java collections library.
The same could be said of many build systems, I suppose.
You are not supposed to write actual build logic inside gradle though, as many people do. It is supposed to be a DSL for build scripts the same way maven is, with slight ease of use scripting.
Couldn't agree more — a hard requirement for a configuration language is that it must be inert.
Under some circumstances, configuration needs to be generated dynamically so you need a programming language — and TypeScript would be fine for that. But a "perfect" configuration format isn't perfect if it can run arbitrary code.
Many years ago, I had the idea of using Ruby as a configuration language. It lends itself nicely to representing tree structures and has some convenient features like ranges. It all worked well as long as we stuck to the subset of Ruby I intended.
One day one of the other developers introduced a change to the configuration that had a side effect. It made sense; after all, the configuration is read at startup, so surely it makes sense to put code there that you want to have run at startup? You can imagine my surprise when I made change that read the configuration again after startup.
Configuration should be purely functional and idempotent, with no side effects, or as you say, inert.
Then should we should do what the industry does instead — start with a flat configuration language like YAML and then add modules, variables and scopes, iteration...
And what do we do when we want the Cartesian product of two sets? Ahem Terraform.
* They can be non-deterministic (do a different thing each time you run them).
* They can be non-hermetic (access stuff in the environment you don't know about).
* They can do naughty security things.
* You can't present GUIs of them because they aren't declarative.
All but the last one don't exclude programming languages. Here's an interesting project to make hermetic deterministic Javascript (Typescript support is planned):
For the sorts of places where you don't have a GUI for the settings anyway (which is the common case) I think it makes loads of sense. It beats making the kind of declarative programming languages you see in YAML files.
Using pure programming languages for configuration is fine. The trick is that the config "program" should basically just output the final - static - configuration, and set of inputs should be very strictly controlled (i.e. no opening random files etc).
Using static file formats for configuration is wishful thinking for projects that are perfectly specified on day 1 and whose requirements never, ever change.
I'd rather choose a bad idea than having to reimplement a Turing-complete programming language in YAML a few years down the line.
Heaven help you if you cave into temptation and think it's a good idea to dynamically calculate this string with code, like to track varying install location and such.
The line between configuration and code is always a little blurry. There are some projects where using a .js file for the "configuration" of another operation is incredibly useful. TS makes even more sense in many of those situations. Lots of configuration files require pre-processing already, like merging shell env vars in with textutil or something like that. So, I wouldn't dismiss this opinion outright.
I feel like the AWS CDK is a great example of this working well. Rather than puzzling over what config options are valid, I get autocomplete and compiler errors when I insist on using an incorrect option. You also get stuff like deprecation warnings showing up in your editor.
It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
Great little writeup ! After mangling YAML, HCL, JSON for years as an ops engineer, I have come to the same realisation. In fact, I have put this into practice in production pipelines by using: jkcfg[1] for the last couple of years. Two data points: 1. Zero developer support contract rate around YAML syntax and templating issues 2. High number of contributions in our private typescript configuration library from developers. Using typescript as an ops frontend has made operations a lot more approachable to folks.
Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration.
I’m not on board with using a language that only has a single implementation and is complex enough that implementing others is difficult.
Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
CUE does look simple enough to be implemented in every language. I wonder if there's anything preventing that. Perhaps there's space for a CUE-like language with an even simpler spec.
TypeScript has at least 2 implementations: tsc and Deno, but I take your point and agree. That said, most languages probably have a JS runtime, so a basic implementation would be ts->js->interpreter. The ts->js is presumably pretty easy.
Geniuenly curious, what downsides do you see with a single, reference implementation? I come from C++ world where we have several competing compilers, each slightly different from the other, and all this incurs a non-negligible maintenance cost while providing very little benefit in my eyes.
The problem is not a single, reference implementation, the problem is how hard it is to implement another one. "Implementing" JSON is relatively easy, most languages have a JSON library. Most people could probably write a JSON library, not a perfect one, but that would mostly work. And probably do it in a reasonable amount of time. All of that isn't true for Typescript.
Ignoring S-expr based configs for a moment, of the configuration languages, JSON parsing is probably the most straightforward. YAML is a superset of JSON, TOML is probably on par, TS and JS are obviously more complex than JSON, and XML is...XML. All of that is to say, I'd rather write a JSON parser than any of the above.
I don't know of any notable S-expr based config languages, probably because they are so easy to parse, people just roll their own, leading to balkanization.
Windows INI format is going to be easier than any of those. You can parse it in a couple lines in a bash script.
I personally find it silly to need more than that. YAML, JSON, etc. What are we even doing here, besides trying to make life more difficult? Are we doing configuration or are we doing plugins and calling it configuration?
We'd all be better off if we just admitted we really want an extension language and went with embedded Lua or Scheme. With Dockerfile and others you have to run a separate lint program, assuming anyone even bothers doing that. At least with a proper language you would have built-in robustness. Most of these configuration formats/DSLs are half-assed and unnecessary anyway.
As Zababa said, it is more about how easy it is to write another implementation, because config files need to be read in another language than (Typescript in this case).
The easier a config language (syntactically, but also in terms of capabilities), the easier it is to write a parser in Go, and Rust and Java etc. All of this particularly matters for libraries or in larger codebases that are multi-lingual.
Using starlark here because I'm most familiar with it.
Starlark's `load()` statement is "find the file at this exact literal path from the workspace root" (no relative paths allowed either). Typescript/Javascript's import is much more complicated. If you encounter an error in Starlark, you fail right there, no exception handling to implement.
Starlark doesn't allow recursion, has an extremely limited set of built-in types and functions, and that means it is very easy to reason about, and even a naive interpreter is pretty fast at evaluating it. I'm also reasonably certain about writing a simple interpreter for it in a week if I had to port it to a new language.
Of course, I view configuration languages on a continuum.
I'd definitely want a pomodoro timer to just use a .ini file.
TOML/YAML is great for where you are just going to have a bunch of plain type assignments. No JSON, because comments.
Once you want to write larger configurations, and you need to provide APIs to the user in the language, I'd pick Starlark/Pystachio because they are still intentionally restrictive and Python is one of the most popular languages out there.
At this point, you are probably trying to configure something like a large build system (Bazel) or some kind of deployment cluster (say Kubernetes) and you can start providing all sorts of utility libraries (either via bindings to your programming language, or as Starlark libraries). But due to their restricted nature you can still do all sorts of bounded analysis of the evaluation. You can remove things like random() because it's a bad idea.
Yes there are use cases where you want a full blown language to configure something, but those are extremely rare and at that point I think you are no longer "configuring" but just "programming". I very much prefer to keep these 2 things separate.
If you can afford to wear only one hat, then sure, use one special language for configuration. But I can't, I am the entire infra department and I still need to do product and framework work. Not having to switch context, language wise, is a very nice thing to have.
Typescript has a lot of punctuation and thus it's not really human-editable. I'd prefer a ini-style format or dockerfile style (with one directive per line of text, clearly identified by a single uppercase word). Moreover, these formats are so easy to parse that you do not even need a library to do so.
Except there’s the ini where strings need to be quoted and the one where they don’t. The one where white space is allowed after the equal sign and the one where it’s not. The one where keys are case sensitive and the one where they’re not.
LOCALE en-US
TIMEZONE America/New_York
LOGLEVEL info
ENVIRONMENT local
The first version has 19 useless punctuation symbols and a few spurious words that are not related to the configuration itself, but are constraints of the typescript implementation. The second version is much simpler, easier to write and to parse using trivial string processing, without depending on any third-party library.
It is beyond me how could any serious engineer honestly find any advantage in the first version.
Well, if you want nesting on your map then you need something slightly more complicated than your second version, but I will grant that you could do something like:
I have used TypeScript for config in a few projects, with node-config[1], and it’s substantially better IME than JSON/YAML/etc. Yeah it’s unconventional to write configs in a general purpose language, and I understand why that would be undesirable. But having configs validated at compile time is great at preventing bugs.
I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic.
As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
Sorta. I evaluated it a long time ago and decided against ever touching it due to its remote import functionality. Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
> Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc.
Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
Dhall is awesome. Only downside is that the included formatter doesn't preserve comments. Super well-designed and with interesting features like semantic hashing.
The fact that it is compiled does not mean you need to run it through an additional step in your build process. In practice, it would just mean that you include TypeScript in your package, which can either be used as an executable or as a library. It can be completely transparent to the user.
You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
> To use this approach today requires that your project also be written in Typescript. It also means you’ll need to forgo a separate flat-file for your configuration.
You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
> So what use is a configuration file if I need to run it through the build pipeline?
Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library.
> The point of having configuration is to gain the ability to configure software without rebuilding in my mind.
There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
I think we got here because everyone's mixed up describing a state with describing the process to describe a state. Or, configuration versus scripts that apply configuration. Most of the popularly config-as-code systems mix the two up, including Ansible, my favorite one. You end up with "config files" that are, all at once in the same file: 1) configuration, 2) templates for configuration, and 3) programs that modify config when they're run.
I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh.
So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
You know what we don't need? Yet Another Configuration Format! Also, don't tell me I can't add comments to JSON because I sure can - add it as a field! If a comment is that important then it's important enough to transfer along with the dataset and durably persist.
For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules.
Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function.
I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
I think parsing YAML or JSON into typed structures is the easier way to go. I e.g. do that in Golang using a little form validation and coercion library I've written. The end result is a nested, strongly typed data structure. Here's an example: https://github.com/iris-connect/eps/blob/master/settings.go (the accompanying form validation configuration: https://github.com/iris-connect/eps/blob/master/forms/settin...). Hashicorp has some libraries that do similar stuff, I wrote my own though since I need it quite often and wanted to have full control over it. The library is just a few hundred lines of Golang code (https://github.com/kiprotect/go-helpers/tree/master/forms). Undocumented as of now as it's nothing I want to explicitly share with the world.
In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
I've done a similar thing with Scala in the past - ship the compiler jar and then compile your configuration (which is just an abstract class implementation) as part of the startup process.
How to add comments in json or TypeScript config files? Commenting is very much needed for any manually written code or config we keep in repositories.
TypeScript is limiting for describing schema in my experience. Try requiring an array of certain length, for example.
Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage.
To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
Dhall is too unfamiliar for most developers, IMO. Developers don’t want to have to grok an entirely new syntax just for config. I’m personally surprised no one has added type annotations to Starlark yet.
I meant length that depends on another type, not length known beforehand (e.g., if N widgets are defined somewhere, [up to] N elements can exist somewhere else, or configuration should not compile). Naturally, this had to involve generics (similar to [0], except I don’t recall why but the `length` property workaround didn’t work or didn’t provide sufficient type safety in my use case). Eventually, after reading workarounds in a related issue[1], I gave up. TypeScript’s type system is great for its intended use, but when applied to expressing schema constraints it quickly falls short and runtime code execution is required to validate a given structure with all the corresponding drawbacks.
Haven’t had much exposure to other statically typed languages recently, so can’t say off the top of my head. Intuitively it looks like something modern generics should allow easily though, doesn’t it?
As to Dhall, for example, I could validate such a constraint (and more) using a function I can be reasonably sure is safe and won’t halt (i.e. without all the drawbacks of TS and other general-purpose languages).
Probably important to mention that at the moment, TypeScript as a configuration format means anyone who writes a configuration blob for you has arbitrary code execution. This is rarely an issue in practice, though.
If you use it by running a V8 isolate that transforms the exported config into JSON, then sure. But TS-based config is largely for people using TS elsewhere. And in that case:
// this returns in 0 ms
export const config: any = {
get value() {
while (true) {}
}
} as const;
// accessing config.value runs forever
console.log(config.value);
For a less unrealistic perspective, if you write enough configuration in a Turing-complete language, you will eventually be tempted to use the full power of the language. You may find yourself having to write tests for the config, which might be e.g. parsing other files, making HTTP requests, querying the filesystem/OS and more. If you're doing that, you are no longer really writing configuration.
The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format.
ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
I would like to see an option to whitelist syntax/types/keywords in Typescript. Allowing only const, dicts, arrays, assignments (but not while). It could just abort if something other than the whitelisted appears.
It could get rid of most of this problems, and not allow network or infinite looping (or file access, and others). Of course it would not be perfect, but could be good enough.
Agreed. It's a matter of discipline. It is a a balancing act between:
1. A constrained specification where its difficult(but not impossible) to shoot yourself in the foot.
2. Devops: as in equitable participation of operators and developers in maintaining configuration on a build it, run it basis.
Languages like skylark/starlark, dhall, cuelang, jsonnet promise a constrained spec(for operators) while being expressive enough(for developers). But after using skylark, jsonnet and dhall in production, personally I haven't seen enough adoption in terms of developer contribution so the promise of expressiveness which is attractive enough for developers hasn't been true for me.
At my current workplace, we have taken a middle approach. We use typescript to generate/emit yaml/json and use "official" validators to validate this configuration. The developers get a familiar and "blessed" frontend to the configuration while the operators can still enforce schema validation, resources and security policy on the generated config.
But yes, if one could import this base typescript package directly in their application, and generate the configuration which can be then picked up by the operator pipeline and isapplied to the infrastructure. That would be bad. Really bad.
Be careful with terms like "directly" as people seem to get very confused when talking about Deno. "Transparently" may be more accurate.
My understanding of the situation right now is that by default, Deno runs TS files by transpiling using SWC, which strips types without checking them, and then sends the result to V8. Optionally you can enable type checking which invokes TSC (with no emission) before SWC. All this is done "magically" from the user's point of view.
Great idea.
And because you need many constants that often change in such a configuration file you could put that all in another file and include that in the 'real' configuration file. Why not call that, hmm, `config_config.txt`. But as plain text is a bit hard to parse, we could give that file some structure, so that it is like, well, another markup language, just without markup and easier to read and write.
147 comments
[ 3.2 ms ] story [ 204 ms ] threadThe reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability.
If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum.
And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
I also found that the user documentation was just a skosh above useless. It felt like it was more concerned with not being wrong than being helpful. Reminiscent of older javadocs in the java collections library.
The same could be said of many build systems, I suppose.
Under some circumstances, configuration needs to be generated dynamically so you need a programming language — and TypeScript would be fine for that. But a "perfect" configuration format isn't perfect if it can run arbitrary code.
One day one of the other developers introduced a change to the configuration that had a side effect. It made sense; after all, the configuration is read at startup, so surely it makes sense to put code there that you want to have run at startup? You can imagine my surprise when I made change that read the configuration again after startup.
Configuration should be purely functional and idempotent, with no side effects, or as you say, inert.
And what do we do when we want the Cartesian product of two sets? Ahem Terraform.
* They can be non-deterministic (do a different thing each time you run them).
* They can be non-hermetic (access stuff in the environment you don't know about).
* They can do naughty security things.
* You can't present GUIs of them because they aren't declarative.
All but the last one don't exclude programming languages. Here's an interesting project to make hermetic deterministic Javascript (Typescript support is planned):
https://github.com/jkcfg/jk
For the sorts of places where you don't have a GUI for the settings anyway (which is the common case) I think it makes loads of sense. It beats making the kind of declarative programming languages you see in YAML files.
- types if you want them
- IDEs/tooling/a kind of UI-out-of-the box
- often other useful toolchain programs for packaging, management, etc
- a well defined syntax
- and the big bertha for REALLY COMPLICATED config: autocomplete in your tool/ide/etc
- and yeah, turing complete dropouts, meta configuration frameworks for generating config
I'd rather choose a bad idea than having to reimplement a Turing-complete programming language in YAML a few years down the line.
Where execute is to sequence a series of effects that are described in a file.
That's not arbitrary code. That's my compiled, tested and verified code.
It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.
Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration.
[1] https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg
Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
[0] https://cuelang.org/
And they absolutely use TS for the std library, just not in the runtime itself
What exactly do you mean by this?
> And they absolutely use TS for the std library, just not in the runtime itself
Yes, internally. The code that actually makes up deno itself is just plain Javascript.
And JSON parsing is a minefield. http://seriot.ch/projects/parsing_json.html
I don't know of any notable S-expr based config languages, probably because they are so easy to parse, people just roll their own, leading to balkanization.
I personally find it silly to need more than that. YAML, JSON, etc. What are we even doing here, besides trying to make life more difficult? Are we doing configuration or are we doing plugins and calling it configuration?
We'd all be better off if we just admitted we really want an extension language and went with embedded Lua or Scheme. With Dockerfile and others you have to run a separate lint program, assuming anyone even bothers doing that. At least with a proper language you would have built-in robustness. Most of these configuration formats/DSLs are half-assed and unnecessary anyway.
The easier a config language (syntactically, but also in terms of capabilities), the easier it is to write a parser in Go, and Rust and Java etc. All of this particularly matters for libraries or in larger codebases that are multi-lingual.
Using starlark here because I'm most familiar with it.
Starlark's `load()` statement is "find the file at this exact literal path from the workspace root" (no relative paths allowed either). Typescript/Javascript's import is much more complicated. If you encounter an error in Starlark, you fail right there, no exception handling to implement.
Starlark doesn't allow recursion, has an extremely limited set of built-in types and functions, and that means it is very easy to reason about, and even a naive interpreter is pretty fast at evaluating it. I'm also reasonably certain about writing a simple interpreter for it in a week if I had to port it to a new language.
Of course, I view configuration languages on a continuum.
I'd definitely want a pomodoro timer to just use a .ini file. TOML/YAML is great for where you are just going to have a bunch of plain type assignments. No JSON, because comments.
Once you want to write larger configurations, and you need to provide APIs to the user in the language, I'd pick Starlark/Pystachio because they are still intentionally restrictive and Python is one of the most popular languages out there. At this point, you are probably trying to configure something like a large build system (Bazel) or some kind of deployment cluster (say Kubernetes) and you can start providing all sorts of utility libraries (either via bindings to your programming language, or as Starlark libraries). But due to their restricted nature you can still do all sorts of bounded analysis of the evaluation. You can remove things like random() because it's a bad idea.
Yes there are use cases where you want a full blown language to configure something, but those are extremely rare and at that point I think you are no longer "configuring" but just "programming". I very much prefer to keep these 2 things separate.
TOML cleans up most of that, though.
This is the proposal of TFA:
And this is how simple it could be: The first version has 19 useless punctuation symbols and a few spurious words that are not related to the configuration itself, but are constraints of the typescript implementation. The second version is much simpler, easier to write and to parse using trivial string processing, without depending on any third-party library.It is beyond me how could any serious engineer honestly find any advantage in the first version.
I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic.
1: https://github.com/lorenwest/node-config
As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"
Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc.
Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...
In practice I think it works fairly well.
The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.
You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
Yes.
> So what use is a configuration file if I need to run it through the build pipeline?
Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library.
> The point of having configuration is to gain the ability to configure software without rebuilding in my mind.
There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh.
So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules.
Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function.
I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage.
To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
[number, number, number]
[0] https://stackoverflow.com/q/56034226/247441
[1] https://github.com/microsoft/TypeScript/issues/26223
Is that a feature of other languages?
As to Dhall, for example, I could validate such a constraint (and more) using a function I can be reasonably sure is safe and won’t halt (i.e. without all the drawbacks of TS and other general-purpose languages).
Here's an example: https://github.com/jkcfg/jk
That beats having to learn an entirely new language.
It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
[0] https://www.tcl.tk/man/tcl8.6/TclCmd/safe.html
[1] https://www.tcl.tk/man/tcl8.6/TclCmd/interp.html#M48
The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format.
ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
I would like to see an option to whitelist syntax/types/keywords in Typescript. Allowing only const, dicts, arrays, assignments (but not while). It could just abort if something other than the whitelisted appears.
It could get rid of most of this problems, and not allow network or infinite looping (or file access, and others). Of course it would not be perfect, but could be good enough.
1. A constrained specification where its difficult(but not impossible) to shoot yourself in the foot.
2. Devops: as in equitable participation of operators and developers in maintaining configuration on a build it, run it basis.
Languages like skylark/starlark, dhall, cuelang, jsonnet promise a constrained spec(for operators) while being expressive enough(for developers). But after using skylark, jsonnet and dhall in production, personally I haven't seen enough adoption in terms of developer contribution so the promise of expressiveness which is attractive enough for developers hasn't been true for me.
At my current workplace, we have taken a middle approach. We use typescript to generate/emit yaml/json and use "official" validators to validate this configuration. The developers get a familiar and "blessed" frontend to the configuration while the operators can still enforce schema validation, resources and security policy on the generated config.
But yes, if one could import this base typescript package directly in their application, and generate the configuration which can be then picked up by the operator pipeline and isapplied to the infrastructure. That would be bad. Really bad.
It also runs Typescript directly.
My understanding of the situation right now is that by default, Deno runs TS files by transpiling using SWC, which strips types without checking them, and then sends the result to V8. Optionally you can enable type checking which invokes TSC (with no emission) before SWC. All this is done "magically" from the user's point of view.