I still think what made Docker popular where other tools failed was its very watered down imperative style. I was always a little surprised that swarm and other tools for orchestrating containers look absolutely nothing like a dockerfile.
I suppose if you push things to the extreme, there are two ways to do it:
1) Prolog/constraint logic programming.
2) Macro recording and code/config generation.
Both have their upsides and downside. Our current Unix configuration model isn't designed for transparent event-sourcing style which macro recorders require. Even Windows with its Registry is half-baked at best. A lot of side effects are not deterministic. What happens if you try to implement Excel-style macro recording is that you end up with a lot of broken generated code due to missing knowledge or specific side effects that can't be reliably triggered. This requires an AGI to solve.
On the other hand, using Prolog works for high level, very specific stuff like firewall configs. But just like automated proof solvers such as Coq, Lean, you need a strong "standard library" of working configs and bindings to get started otherwise you will be starting from scratch everytime. This is a weakness that's going to be difficult to overcome. There aren't many companies doing developer tooling and integrations that are successful, HashiCorp and Jetbrains being the exception, not the norm. Docker may be an unicorn on paper but it still struggles to turn a profit.
Both of the above methods can be successful if a major corporation is willing to throw money at it. Otherwise we will be stuck with whatever software FAANG decides to release because the developer tooling industry is quite cruel to smaller players. Flynn, Deis etc. Behind every Heroku lies countless dead startups.
I really like k8s itself and I think these yaml files are a great API layer, but the way people produce these files nowadays (templating systems and a lot of glue like helm) is a joke and in a real, even mildy complex environment becomes a terrible mess really quickly. What I really want is a typed programming language, where using normal code we describe how to produce all these yaml-described k8s resources using whatever abstractions make sense in our business.
I still haven't got to trying it. I'm a bit suspicious, because it look like a SaaS kind of thing, while I was looking just for an open source tool to write some code that will spit out bunch of `yaml` files to feed to `kubectl`.
The runner is free, as is the tool chain. One of my employers has paid money for the SaaS state management; two others have not. We’ve gotten by just fine.
I am also suspicious as well. The biggest turn off for me initially was the fact that you had to login to use even the free tool. Apparently though this has been addressed somewhat overtime. (https://www.reddit.com/r/devops/comments/bcdwsn/pulumi/)
> What I really want is a typed programming language, where using normal code we describe how to produce all these yaml-described k8s resources using whatever abstractions make sense in our business.
This is more or less exactly what we’re working on at Pulumi. Using a language like TypeScript to describe Kubernetes infrastructure provides a nice mix of simple JSON-like resource definition, strong typing, and the ability to use variables, loops, functions, classes, packages, test frameworks, and more from an expressive general purpose programming language. You still have access to the full fidelity of the raw Kubernetes API, but also all the tools you need to apply proper software engineering on top of it.
"We're"? Oh. So I have one question - what's the deal with Pricing then? What's the line exactly between "take it for free" and "pay us some well deserved money for our hard work"?
Anyway - from what you're saying, that's exactly what I'm looking for.
Huh, why build an entirely proprietary language for this? There are existing toolchains that bend existing rich configuration languages, like Nix [0] or Dhall [1], to produce Kubernetes object definitions. In this way, not only can existing libraries be leveraged, but no new semantics have to be learned, and no third party needs their palms crossed.
This had to be pointed out; Pulumi's docs repeat the adjective "real", as if its competitors aren't real. "real code", "real package management" [2], "real languages" [3], "real programming model" [4]... Nix and Dhall are quite real, though. So is this an incomplete survey, or are Pulumi's docs unrealistically self-aggrandizing? TBH even just this one paragraph of yours is pretty tough to stomach, as if loops and functions in a configuration language were a novel concept.
I would imagine that a "real programming language" here means not only any Turing-complete language but also a language that has been consistently at the top 10 most popular programming languages in the world for a while (which javascript is, and typescript is its superset).
I would hope that we can do better than that. Both Nix and Dhall are designed for readability in the limited niche of configuration management, and their authors considered that design choice to be important enough on its own to merit a new language. Dhall is explicitly not Turing-complete.
If you're willing to relax the "Turing-complete" part, and do a bit of final assembly yourself, you might try https://github.com/stripe/skycfg which uses the Python-like Starlark language.
> What I really want is a typed programming language, where using normal code we describe how to produce all these yaml-described k8s resources using whatever abstractions make sense in our business
I had high hopes for CDK but after following the recommended path for a not very complex deployment and ending up with 3x more code than terraform, I had to declare it unfit for mass consumption. I spoken with my rep about it and he tried really hard to steer away from it because they know it's just not there yet. It's better than CFN but only because it's not plain yaml. As it matures, if they can get community buy in, I'm sure it'll be good enough but I'd leave it alone for now.
It is, but it isn't very pleasant to use. It's the difference between Sublime Text and Notepad++. A curmudgeon software engineer may argue that the former is lipstick on a pig because the latter exists. But developer UX is important, the difficult part is getting them to pay for it (just google the amount of Sublime Text piracy posts on GitHub).
Well, I'd be pleasantly surprised to know it is Turing complete. For example, TeX is (IMHO) very unpleasant to use, but due to it being Turing complete, its abstraction LaTeX is much easier.
Would you like to expand on your view? I’ve used terraform a lot and never had any issues with the tooling. But then again it’s just a pretty simple CLI/config setup? Do you mean you’re looking for graphical interfaces to write config files?
Personally, HCL is what bothers me. Yes, it’s infrastructure-as-code... but that code is in a niche proprietary language. It gets unpleasant as soon as you go beyond trivial static config.
Pulumi and AWS CDK are taking what I think is a better approach: use your favourite general-purpose language to build the declarative infrastructure definition.
> "A System with 1000+ Microservices at Planet Scale described As Code 100% Declaratively"
There's a fundamental problem here that I don't believe a (taking Pulumi at face value) better config language solves: you are working with an inherently complex system, and configs are only one part of that complexity.
Doesn't matter if you have 1k+ microservices, a monolith with 1k+ RPC services, or something in between - at that scale, you have accumulated such complexity in the system that configs are just one facet of the problem.
I do think it's an interesting premise (but I wonder if they've thought about the zip bomb problem, which has persisted across many, many new technologies, including yaml - at some level, you want to try to make guarantees about config parsing runtime requirements - cpu, memory, disk). An idea that's bounced around my head before is letting folks define small programs - Java, Python, whatever - to generate configs, and use static analysis to encourage hermeticity/reproducibility/etc (e.g. disallow random, disallow concurrency, no envvar deps).
A similar, but related problem: people very frequently write configs such that you could change settings x, y, z using this one line, often for the sake of comprehensibility/readability. But we have no way of writing tests for this- all you can do is hope whoever does the code review keeps that in mind.
I think the real problem we have with infrastructure is one of evolution, abstractions and leaky abstractions.
First, infrastructure is not a static thing, it changes over time in response to code updates, environmental load (autoscaling), etc. And the details here matter. How should new code be rolled out (blue/green, how fast, what are the quality metrics, etc.). What should happen to the old instances while the new stuff is rolling out? When should they be killed?
What we really need is a good way to declaratively specify what to do, and how the components in that declarative specification should evolve as new stuff is rolled in, load changes, etc. We need a good way to specify the control plane for our infrastructure. adaptjs (adaptjs.org) is our TypeScript-based open source project to provide this. It is still early stage, but I think it is pretty promising.
The second issue is one of abstraction. All these things are complex and often need customization to suit the particular application challenges. Using templates, CDK, Pulumi, or anything else to generate YAML is fine, but as the article points out, you have no insight into how your specification got realized into the low-level infrastructure.
Moreover, you have to care since you need to know the low-level implementation to debug, understand cost, analyze logs, etc. Adapt has some machinery so that you don't always need to understand the low-level implementation for this stuff, but since it is still early, that part of the system needs work before it is ready for prime time.
31 comments
[ 4.5 ms ] story [ 83.1 ms ] thread1) Prolog/constraint logic programming.
2) Macro recording and code/config generation.
Both have their upsides and downside. Our current Unix configuration model isn't designed for transparent event-sourcing style which macro recorders require. Even Windows with its Registry is half-baked at best. A lot of side effects are not deterministic. What happens if you try to implement Excel-style macro recording is that you end up with a lot of broken generated code due to missing knowledge or specific side effects that can't be reliably triggered. This requires an AGI to solve.
On the other hand, using Prolog works for high level, very specific stuff like firewall configs. But just like automated proof solvers such as Coq, Lean, you need a strong "standard library" of working configs and bindings to get started otherwise you will be starting from scratch everytime. This is a weakness that's going to be difficult to overcome. There aren't many companies doing developer tooling and integrations that are successful, HashiCorp and Jetbrains being the exception, not the norm. Docker may be an unicorn on paper but it still struggles to turn a profit.
Both of the above methods can be successful if a major corporation is willing to throw money at it. Otherwise we will be stuck with whatever software FAANG decides to release because the developer tooling industry is quite cruel to smaller players. Flynn, Deis etc. Behind every Heroku lies countless dead startups.
Twitter thread I had about it a while ago: https://twitter.com/dpc_pw/status/1182022825361625088
This is more or less exactly what we’re working on at Pulumi. Using a language like TypeScript to describe Kubernetes infrastructure provides a nice mix of simple JSON-like resource definition, strong typing, and the ability to use variables, loops, functions, classes, packages, test frameworks, and more from an expressive general purpose programming language. You still have access to the full fidelity of the raw Kubernetes API, but also all the tools you need to apply proper software engineering on top of it.
https://www.pulumi.com/kubernetes/
Anyway - from what you're saying, that's exactly what I'm looking for.
This had to be pointed out; Pulumi's docs repeat the adjective "real", as if its competitors aren't real. "real code", "real package management" [2], "real languages" [3], "real programming model" [4]... Nix and Dhall are quite real, though. So is this an incomplete survey, or are Pulumi's docs unrealistically self-aggrandizing? TBH even just this one paragraph of yours is pretty tough to stomach, as if loops and functions in a configuration language were a novel concept.
[0] https://github.com/xtruder/kubenix
[1] https://github.com/dhall-lang/dhall-kubernetes
[2] https://www.pulumi.com/docs/intro/vs/k8s_yaml_dsls/
[3] https://www.pulumi.com/docs/intro/vs/chef_puppet_etc/
[4] https://www.pulumi.com/docs/intro/vs/custom/
Cruise built a more complex DSL on top of it: https://github.com/cruise-automation/isopod
For what it's worth, that's what AWS cdk is. Not for k8s but for AWS CloudFormation yaml templates. https://docs.aws.amazon.com/cdk/latest/guide/home.html
Even is you make infrastrcture coding as simple as SQL DDL, you have still said naught of the specific application's schema/design.
Once you are past a dirt-simple, low-traffic LAMP app, I'd expect all bets are off.
(edit: typo)
Pulumi and AWS CDK are taking what I think is a better approach: use your favourite general-purpose language to build the declarative infrastructure definition.
There's a fundamental problem here that I don't believe a (taking Pulumi at face value) better config language solves: you are working with an inherently complex system, and configs are only one part of that complexity.
Doesn't matter if you have 1k+ microservices, a monolith with 1k+ RPC services, or something in between - at that scale, you have accumulated such complexity in the system that configs are just one facet of the problem.
I do think it's an interesting premise (but I wonder if they've thought about the zip bomb problem, which has persisted across many, many new technologies, including yaml - at some level, you want to try to make guarantees about config parsing runtime requirements - cpu, memory, disk). An idea that's bounced around my head before is letting folks define small programs - Java, Python, whatever - to generate configs, and use static analysis to encourage hermeticity/reproducibility/etc (e.g. disallow random, disallow concurrency, no envvar deps).
A similar, but related problem: people very frequently write configs such that you could change settings x, y, z using this one line, often for the sake of comprehensibility/readability. But we have no way of writing tests for this- all you can do is hope whoever does the code review keeps that in mind.
ansible & friends fell well short.
biggest peeve is vscode allows you to programmatically install extensions yet firefox doesn't.
something something nixos but i'd like that for fedora/ubuntu etc.
I think the real problem we have with infrastructure is one of evolution, abstractions and leaky abstractions.
First, infrastructure is not a static thing, it changes over time in response to code updates, environmental load (autoscaling), etc. And the details here matter. How should new code be rolled out (blue/green, how fast, what are the quality metrics, etc.). What should happen to the old instances while the new stuff is rolling out? When should they be killed?
What we really need is a good way to declaratively specify what to do, and how the components in that declarative specification should evolve as new stuff is rolled in, load changes, etc. We need a good way to specify the control plane for our infrastructure. adaptjs (adaptjs.org) is our TypeScript-based open source project to provide this. It is still early stage, but I think it is pretty promising.
The second issue is one of abstraction. All these things are complex and often need customization to suit the particular application challenges. Using templates, CDK, Pulumi, or anything else to generate YAML is fine, but as the article points out, you have no insight into how your specification got realized into the low-level infrastructure.
Moreover, you have to care since you need to know the low-level implementation to debug, understand cost, analyze logs, etc. Adapt has some machinery so that you don't always need to understand the low-level implementation for this stuff, but since it is still early, that part of the system needs work before it is ready for prime time.