Pkl is a new language for describing configuration. It blends together the declarative nature of static formats like YAML or JSON, with the expressiveness, safety, and tooling of a general purpose programming language.
I was a big believer in Helm for generating Kubernetes resources until I looked up and saw that we had created an impossible-to-validate, impossible-to-reason-about DSL in our values.yaml and that's when I realized we were at the end of our rope with Helm. We switched to Pkl for our Kubernetes resource generation -- it's delightful to maintain and reason about our deployments before execution time :)
Pkl was one of the best internal tools at Apple, and it’s so good to see it finally getting open sourced.
My team migrated several kloc k8s configuration to pkl with great success. Internally we used to write alert definitions in pkl and it would generate configuration for 2 different monitoring tools, a pretty static documentation site and link it all together nicely.
Would gladly recommend this to anyone and I’m excited to be able to use this again.
Was about to ask if you had k8s api models available internally, and that someone should create some tool to generate that from the spec. But turns out it already exists in the open!
Python isn't oriented around defining and validating data.
For example, something like: "this number is be between 1 and 10" means you have to come up with a novel way to add validation, because it isn't built into the language.
Also, Pkl is meant to be usable everywhere. General purpose languages tend to be tied to their own ecosystem--imagine telling a Go developer that they need to install Python, set up virtualenv, and run pip install so they can configure their application. We'd like Pkl to be a simple tool that can be brought anywhere, and easily integrated into any system.
> For example, something like: "this number is be between 1 and 10" means you have to come up with a novel way to add validation, because it isn't built into the language.
No need for a novel mechanism - there are plenty of available solutions to add validation to python
> imagine telling a Go developer that they need to install Python
Pkl doesn't come preinstalled on machines - so you'll have to install it as well
> set up virtualenv, and run pip install so they can configure their application
This is the real friction point, but is it a bigger friction point than having to adopt yet another DSL?
> This is the real friction point, but is it a bigger friction point than having to adopt yet another DSL?
Yesterday, I created a virtualenv, then ran `pip install`, only to see it fail. I found out that even `pip --version` was failing. I discovered that running `python -m ensurepip --upgrade` would fix pip. It did fix pip, but `pip install` still didn't work properly. I figured out that pip was reporting a different version of python than the one virtualenv is using. Running `python -m pip install --upgrade pip` upgraded pip, which should have been accomplished with the previous ensurepip command. Finally, everything was working properly.
I experienced these problems after years of experience in python. To answer the question. Yes, it's worth adopting yet another DSL than using python.
Yes, I had activated the virtual environment. Thanks for the guide. It works without problems. I had used `virtualenv venv -p /usr/local/bin/python3.12` in my setup. Yours seems to be a better way.
Probably the infrastructure/packaging mess. That's debatable. If you fix all of them though it's easily worth a new language, as long as it isn't too hard to learn.
This looks very easy. I'd be more hesitant about Dhall or Nix (though obviously Nix comes with even bigger benefits so it might be worth the awkward language).
Because (unless your app is written in Python) you don’t want to start a full-fledged Python run-time to read a config file. Nor do you want all the hassle of trying to ship a Python run-time with your application.
Edit: and moreover, you probably do not want config files that can run arbitrary code with full access to the Python standard library.
Types on the fields is interesting. The examples suggest said types are lost when you serialise to json or xml, though it seems like at least some types should be expressible as schemas.
There’s another repo, the Pkl Pantry, that provides a couple of ready made templates (Schemas) that you can try out: https://github.com/apple/pkl-pantry
I was looking for a translation from a pkl template into an xml schema. It would probably be lossy - I think pkl can express more invariants than xml or json schema can - but still gives some invariants on the data for use downstream of the pkl tooling.
I thought it odd that the language bindings didn't include the most popular langage, Python. In fact, Python seems not to be mentioned at all in the linked page. So I'm wondering, is it
1) Because the developers of Pkl are Python haters
2) Because the developers of Pkl are so overawed by Python that they can't imagine Pkl contributing anything useful to the Python ecosystem
In either case, having suffered so much using Ansible and its excrable YAML scripting, I may use Pkl together with Python.
Probably secret option number 3 of no-one having needed it yet - because the tool is standalone, you can render the required output before running whatever needs to consume it. I’ve certainly used it from Python in that manner - bindings are only required if you want to consume the raw language programmatically from a Python context.
Exactly this. Pkl is most useful as a type-safe configuration language that can output to any other format (already supported, or put together by the user within Pkl). You’ll always get valid JSON, YAML, PLIST, what-have-you as output. This you can then parse in the language/system of your choice.
Certainly language bindings are useful, and if there’s demand likely someone will create them.
All the listed languages are compiled and statically typed. Python is neither. Neither is JavaScript, another popular programming language which is also not listed.
I don't see how the static typing of the bound languages enters into it. Python has type hints which can be enforced by some compilers, if you are so committed to type safety in your config scripts. If you have Pkl output YAML or XML then where is your type safety?
Since "popular" does mean that very many people are using it, I think it would be wise to try to serve the Python community.
Look at the languages that they do support - Go, Swift, Kotlin, Java. These are all robust languages for writing production grade software. That's probably why - the people at Apple using this don't need it for their hacky Python scripts.
I think this intended for compiled languages? If you already have Python or Ruby in your stack you can simply write a little script to generate the required JSON or YAML. I'm not sure you would ever want to add Pkl to the mix in that case?
This reminds me of the idea behind Lua - similarly the original users needed a configuration format which became increasingly sophisticated and at some point the authors realized they needed “proper” programminglanguage constructs.
This is also why Lua is called Lua, the original configuration language was called SOL, for Simple Object Language. It never shipped, by the time the desired code was delivered to Petrobras, it was the first edition of Lua.
The authors have a fun read[0] about the history of the language, for the curious.
> Pkl — pronounced Pickle — is an embeddable configuration language which provides rich support for data templating and validation. It can be used from the command line, integrated in a build pipeline, or embedded in a program. Pkl scales from small to large, simple to complex, ad-hoc to repetitive configuration tasks.
I do like the sound of that. It's always quite tedious to manage configuration in full-stack applications with mixed languages/ecosystems. It seems they already have Pkl plugins for IntelliJ, vscode and neovim and a language server is "coming soon".
Where "format" is used in the lightest way possible. (NEVER try to unpickle anything that was not produced by pickle itself (preferably the exact same version))
JSONSchema covers a lot of various schema needs and YAML is something a lot of developers are comfortable with. I know both of those technologies are not popular here in HN but YAML type-checked and editor-autocomplete-enabled using JSONSchema is a solid choice for configurations in my opinion.
Every time I see YAML used for any configuration I know I’m in a frustrating time. It’s particularly bad for build systems where there the feedback time can be so slow.
true, but as name suggests jsonschema wasn't meant for yaml it's only because yaml is superset of JSON.
the issue is that when you want or need to use something that is outside of JSON spec, like tags all your validation falls apart.
also JSON schea validation is really basic, and while designing configuration format can often mitigate that, it's not very versatile.
another common thing is that sometimes you have to define multiple very similar sections in configuration that cannot be handled with yaml archors, eg I have repeated definition dozen times that changes only in 2 numbers that are deeply in structure and name string, and I need to repeat everything because of that and it's pita to modify all other parameters that need to be kept in sync
therefore I think this format looks really nice, although I'm concerned by loops that can be used there, is there possible to create simple config that takes ages to parse or consume very large amounts of memory?
Jsonschema is still json and yaml is absolutely not comfortable to work with. It’s only enough for simple configs. As soon as you have the urge to use a template you should replace it with something else.
I use jsonnet for templating and transformation and jsonschema for validation. Very happy with this combination. One big reason is that there are lot of libraries and codegen tools to choose from once you have the JSON generation pipeline (including schema generation) down.
Python is able to load Python code at runtime, so one can use Python for configuration. A solution doesn’t need to go looking for problems that are already solved.
People interested in configuring Python software in Python
should look into Starlark.
There are Python bindings for two versions of Starlark:
Go (https://github.com/caketop/python-starlark-go)
and Rust (https://github.com/inducer/starlark-pyo3).
I used python-starlark-go for a time
in a project that ran on x86-64 Linux
and had no problems with it.
(I stopped because my project's configuration turned out simpler than expected,
so I switched to TOML.)
Worth noting that it is specifically CPython that has been called impossible to sandbox.
(2014 discussion: https://news.ycombinator.com/item?id=8280053.)
It may be possible to sandbox PyPy.
PyPy has a sandboxing feature its website calls a "working prototype"
(https://www.pypy.org/features.html#sandboxing).
If someone invested in it—potentially a huge effort—it could plausibly become good enough for configuration.
But, IMO, Starlark is a better choice here
because it was designed for isolation from the start.
If you wanted to invest in Python-as-config-for-Python,
a good use of your time might be improving Starlark for Python.
So true. One of the major config mgt utilities which shall remain nameless, (cough, Ansible, cough), is written in python but created an excrable POS config language build on YAML. At least Ant had the excuse that Java was not suitable for a config language. Will people never learn that building scripting languages on markup languages will inevitably end in tears?
I'm sorry, can someone explain why one would want to translate from one data description language to another (Pkl -> JSON, or whatever)? Why not just write JSON (or whatever) to begin with?
Because Pkl makes it trivial to write templates and transforms. So you can write a Pkl schema that only requires a minimum set of fields, then auto generate a complete configuration.
This is most useful when dealing with tools like k8s where deploying a single application might involve 3-10 separate manifests (Deployment, Service, NetworkPolicy, HttpRoute, Autoscaler etc etc). With Pkl you can easily write a simple template that only requests the minimum needed to define an “app” (e.g. name, namespace, mixins for sidecars) and have Pkl generate all the needed manifests for you.
Really Pkl should be seen as a language for quickly building templating tools like Helm. But with type safety by default, and no need for horrible indent hacks.
Well, Dhall provides something between JSON and a Turing complete language that can make a lot of configuration much quicker to write, if you can hack the functional syntax. Pkl is probably a similar concept.
In general (not just limited to Pkl), I think the advantage is that you get IDE support like autocomplete and compile time checks. Pkl seems to borrow some features from JSONSchema/SHACL for example where one can also add validations like "value must be bigger than 20 and lower than 100" so when you configure a component incorrectly, it can throw a good error message before deployment.
Json lacks comments, templating, evaluation, types, and lots of other features. Having something generating a valid config is great, especially if you're generating multiple configs in different format from the same place. For example being able to configure SSH, nginx and some other services from nixos config is amazing.
Writing raw JSON is very error prone and you have to repeat yourself a lot. I think everyone who has worked with it has had some surprises, I certainly have.
Similar to why do we write Python instead of assembly, or why do relational databases typically have things like datatypes and constraints?
The amount of problems I've had with JSON in my career makes me think almost anything could be better than it. There's so many weird edge cases in the JSON spec that you can hit that it just becomes endless levels of hair pulling.
597 comments
[ 4.8 ms ] story [ 316 ms ] threadI've looked at cdk8s but I hate the tsconfig culture, why can't it be simple like Go?
My team migrated several kloc k8s configuration to pkl with great success. Internally we used to write alert definitions in pkl and it would generate configuration for 2 different monitoring tools, a pretty static documentation site and link it all together nicely.
Would gladly recommend this to anyone and I’m excited to be able to use this again.
https://github.com/apple/pkl-k8s-examples
https://github.com/apple/pkl-k8s-examples/blob/96ba7d415a85c...
It is so sensitive that basic text editing like copy and paste, tab, in/decreasing indent never quite do what I expect in IntelliJ.
I paste parts of yaml into another yaml and it ends up somewhere unpredictable.
For example, something like: "this number is be between 1 and 10" means you have to come up with a novel way to add validation, because it isn't built into the language.
Also, Pkl is meant to be usable everywhere. General purpose languages tend to be tied to their own ecosystem--imagine telling a Go developer that they need to install Python, set up virtualenv, and run pip install so they can configure their application. We'd like Pkl to be a simple tool that can be brought anywhere, and easily integrated into any system.
I'm also incredibly high.
Have a look at PEP 20 – The Zen of Python.
Python is actually horrible at following it, Go doing a much better job.
No need for a novel mechanism - there are plenty of available solutions to add validation to python
> imagine telling a Go developer that they need to install Python
Pkl doesn't come preinstalled on machines - so you'll have to install it as well
> set up virtualenv, and run pip install so they can configure their application
This is the real friction point, but is it a bigger friction point than having to adopt yet another DSL?
Yesterday, I created a virtualenv, then ran `pip install`, only to see it fail. I found out that even `pip --version` was failing. I discovered that running `python -m ensurepip --upgrade` would fix pip. It did fix pip, but `pip install` still didn't work properly. I figured out that pip was reporting a different version of python than the one virtualenv is using. Running `python -m pip install --upgrade pip` upgraded pip, which should have been accomplished with the previous ensurepip command. Finally, everything was working properly.
I experienced these problems after years of experience in python. To answer the question. Yes, it's worth adopting yet another DSL than using python.
For all the flak python gets, dependency setup is pretty simple if you're not flailing around aimlessly
And to think "There should be one-- and preferably only one --obvious way to do it." is in the Zen of Python...
What novel way to add validation in python?
* It's been dragged into static typing kicking and screaming.
* You import the whole Python infrastructure/packaging catastrophe.
* It's not sandboxed.
If you wanted something Python-like you would you Starlark.
This looks very easy. I'd be more hesitant about Dhall or Nix (though obviously Nix comes with even bigger benefits so it might be worth the awkward language).
Edit: and moreover, you probably do not want config files that can run arbitrary code with full access to the Python standard library.
Language reference doesn't mention schemas either https://pkl-lang.org/main/current/language-reference/index.h...
There’s another repo, the Pkl Pantry, that provides a couple of ready made templates (Schemas) that you can try out: https://github.com/apple/pkl-pantry
1) Because the developers of Pkl are Python haters
2) Because the developers of Pkl are so overawed by Python that they can't imagine Pkl contributing anything useful to the Python ecosystem
In either case, having suffered so much using Ansible and its excrable YAML scripting, I may use Pkl together with Python.
Certainly language bindings are useful, and if there’s demand likely someone will create them.
Is it really that deep?
All the listed languages are compiled and statically typed. Python is neither. Neither is JavaScript, another popular programming language which is also not listed.
Since "popular" does mean that very many people are using it, I think it would be wise to try to serve the Python community.
Look at the languages that they do support - Go, Swift, Kotlin, Java. These are all robust languages for writing production grade software. That's probably why - the people at Apple using this don't need it for their hacky Python scripts.
The authors have a fun read[0] about the history of the language, for the curious.
[0]: https://www.lua.org/history.html
I do like the sound of that. It's always quite tedious to manage configuration in full-stack applications with mixed languages/ecosystems. It seems they already have Pkl plugins for IntelliJ, vscode and neovim and a language server is "coming soon".
(is there an xkcd for that?)
https://cdn.statcdn.com/Infographic/images/normal/1438.jpeg
you can click it, it'll open link in your browser where you can read what it refers to.
- mature libs for most languages
- VScode plugin + Jsonschema for auto completion / schema checking
- yamllint to detect the languages footguns
- yq to query, update in place, and format while preserving comments and sorting keys from the CLI
another common thing is that sometimes you have to define multiple very similar sections in configuration that cannot be handled with yaml archors, eg I have repeated definition dozen times that changes only in 2 numbers that are deeply in structure and name string, and I need to repeat everything because of that and it's pita to modify all other parameters that need to be kept in sync
therefore I think this format looks really nice, although I'm concerned by loops that can be used there, is there possible to create simple config that takes ages to parse or consume very large amounts of memory?
(Never used Pkl myself...)
Granted the conveniences of Python syntax for code are mostly lost when trying to express tree structured data, and yaml flips that on its head.
(Mumble, grumble, something about s-expressions...)
But it’s still better than templating yaml.
Worth noting that it is specifically CPython that has been called impossible to sandbox. (2014 discussion: https://news.ycombinator.com/item?id=8280053.) It may be possible to sandbox PyPy. PyPy has a sandboxing feature its website calls a "working prototype" (https://www.pypy.org/features.html#sandboxing). If someone invested in it—potentially a huge effort—it could plausibly become good enough for configuration. But, IMO, Starlark is a better choice here because it was designed for isolation from the start. If you wanted to invest in Python-as-config-for-Python, a good use of your time might be improving Starlark for Python.
For in-house stuff, totally agree, just use the python code itself as the configuration.
This is most useful when dealing with tools like k8s where deploying a single application might involve 3-10 separate manifests (Deployment, Service, NetworkPolicy, HttpRoute, Autoscaler etc etc). With Pkl you can easily write a simple template that only requests the minimum needed to define an “app” (e.g. name, namespace, mixins for sidecars) and have Pkl generate all the needed manifests for you.
Really Pkl should be seen as a language for quickly building templating tools like Helm. But with type safety by default, and no need for horrible indent hacks.
http://dhall-lang.org/
Something like Cuelang will:
- Define the schema that will let you know what you should put inside.
- Allow you to inflate that schema into a full file, providing only the values.
- Will generate always a correct file, with no typo, or format error.
- Will check that the data is correct and tell you if there are any errors.
- Is note theoretically tied to a particular run time or stack.
I'll be looking into cue, but how does it solve that problem ?
https://json-schema.org
Similar to why do we write Python instead of assembly, or why do relational databases typically have things like datatypes and constraints?