Tanka: Our way of deploying to Kubernetes (grafana.com)

Anchors are fairly limited:

- they are bound to a single file. This won’t help you when trying to maintain multiple similar sets of Config

- anchors do not support patching. If you need to change a nested key, you can’t do so without it affecting all other nested keys as well

I found that odd too. Typical examples like https://confluence.atlassian.com/bitbucket/yaml-anchors-9601... are pretty easy to map to their example of why they need repetition.

Which extensions are you talking about? Do you mean CRDs? Then it take plain JSON or you can write a function to generate those.

Exactly like YAML does.

With Tanka you would usually use the functions provided by `k.libsonnet` to generate your manifests.

For CRD's, there are no helpers available, so you either just write the plain manifest as a Jsonnet object (JSON syntax), as YAML and `import` it (gives you an object as well) or even better write some helper functions yourself, publish them as a library on GitHub and make future users happy :D

Interesting, this is similar to jsonnet. Does it support deep merging?

For example, in our case we do `default_deployment.libsonnet` to which we add arbitrary overrides like `default_deployment + dev` or `default_dep + us_central1_overrides` depending on the cluster. And in these case we don't need to pre-specify what can be overriden in the `default_deployment` which makes it really powerful (also can come bite you in the arse though).

Hey we've been using ksonnet since before kustomize existed and we've just rewritten it to be more flexible with much more focus.

Further we use jsonnet a lot (including generating dashboards) and in general found it much more powerful and useful compared to plain YAML. From a customary glance, I can't find if kustomize supports jsonnet.

Take a look into kustomize

https://github.com/kubernetes-sigs/kustomize

Grafana Labs employee here. We've been using jsonnet-bundler as a package manager on top of our Tanka configs. As for some common packages, we open source most of ours[1]. We also provide Tanka-compatible configs for Loki[2].

[1]: https://github.com/grafana/jsonnet-libs

[2]: https://github.com/grafana/loki/tree/master/production/ksonn...

I'd also recommended taking a look at using Terraform[1] to manage Kubernetes resources instead of yaml files and kubectl.

[1] https://www.terraform.io/docs/providers/kubernetes/guides/ge...

There's another jsonnet tool called qbec that can do that - https://qbec.io

(Tanka dev here)

Nobody can neglect the power of helm charts (because so many already exist), so I think Tanka will add support soon.

Even though we are focused on Jsonnet right now, this does not necessarily need to stay that way forever.

Grafana for example is popular because it supports multiple datasources, Tanka should probably do as well (e.g. Jsonnet, CUE, Helm, whatever results in JSON)

kubernetes is the invasion of js devs to the server - wondering when all the neckbeards will wake up and start fighting back

You could use Tanka and Jsonnet to improve this for Grafana dashboards: https://github.com/grafana/grafonnet-lib.

Would that work for you?

Maintainer of jsonnet-bundler, kube-prometheus and some monitoring mixins, that are all based on jsonnet, here:

Currently we're mostly keeping a close look at CUE, but not really using it as of right now. However, during the holiday break I've been trying to get into CUE again and there are some things I need to figure out before being able to tell how to incorporate or replace some of our jsonnet projects with CUE, if we really want that.

Some parts of CUE seem like an obvious improvement to what jsonnet currently offers. So 2020 will be exciting in that regard.

It's still early, but incredibly exciting. I personally am betting on CUE as the "winner" in this space. Its creator is enormously credible. Most of the other configuration languages in this space are directly inspired by his work. So for him to work on something new is significant.

I am also impressed by the clip at which CUE is improving, and how useful it already is, in spite of its relatively young age. It reminds me of Python or Go in its focus on tooling quality, and stability.

Their community slack is very welcoming, too. https://cuelang.slack.com

I am using it for k8s deployment and overall am quite happy that I can safely configure deployments without boilerplate. The k8s integration makes it easy to get started with, including using existing yaml configuration. Certainly though you have to invest some time to learn it and I have discovered a few rough edges.

I haven't seen another appealing solution in the space. I recommend using Kustomize if you can, but it is very limited in what you can do with it. When I looked at dhall-kubernetes it was lacking some crucial defaulting features that are getting integrated into the language now.

Here’s one example, not using any of these “templating turned programing languages” tools:

We build an api and a cli util for these things.

The api takes care of providing sane deployment manifests based on data from a service registry, which is populated using the cli.

Cli can be run non-interactive for use in pipeline and other automations.

The cli and api basically do any config and deplyment related tasks in concert with service discovery (which at this time is consul).

Edit, rant: For automations json just owns yaml in all the ways!

This yaml templating business should stop! I think it’s silly tbh. :)

Our base k8s setup consisted of roughly 40k lines of yaml. Yaml is awesome for 10 lines of user input configuration, not 100, let alone 40k machine printed stuff. What happened here in this brave new DevOps world?!

For integrating k8s deploy into CI / CD, I really like Krane [0]. It has several key features:

  1. A way to templatize YAML (though it could be used with Tanka too - the render command is split from the deploy command for this reason)
  2. Monitors the rollout of resources - it's possible to detect successful and failed deployments more easily.
  3. A way to run one-off commands during deployment.
  4. Uses kubectl under the hood so easy to add to any workflow.

It has other features that I don't use including secrets management.

Net net - I definitely think it could elegantly replace a bunch of bash scripts that do kubectl things.

[0] https://github.com/Shopify/krane

For CI (testing) we use tekton [1] to run tests and pass around artifacts inside k8s. You can kick off and monitor tekton builds with their CLI tool, but we ended up building our own create/monitor/download-artifacts tool for it in ruby. We use an off the shelf CI server to kick that tool off, and it dumps back results the CI server can understand.

One of the things we need to do is elastically scale the number of tasks (basically pods in tekton) that comprise our test suite run. This might be based on cluster utilization or whether it's the master branch. Since we have a single threaded test suite, we hoist parallelism up to the k8s level by breaking apart the tests into partitions each run by its own pod. For this we just render processed and parameterized erb to yaml. Eventually we'll dispense with yaml altogether and programmatically construct resources using a k8s REST api client.

We haven't moved into CD with any of this tooling yet.

[1] https://github.com/tektoncd/pipeline

(Tanka dev here)

I think this highly depends on your deployment process .. do you want full continuous deployment (CI deploying to the cluster)? In this case you could continue to use for example Jenkins to run `tk apply <environment>` on each merged PR.

Another option would be to use an in-cluster CD agent (for example https://fluxcd.io/), which uses Tanka to generate the yaml and applies it. Flux can be used with Tanka, needs some setup though: https://docs.fluxcd.io/en/1.17.0/references/fluxyaml-config-.... I guess we could simplify this in the future.

Feel free to reach out to me on Slack http://slack.raintank.io/ in the #tanka channel :D

I would argue the same about yaml and whitespace being meaningful. Oh you forgot a single space? Your markup is screwed. Xml is much better...

Hi, you won't have much contact with JSON at all.

While Jsonnet is technically a superset, all you will see during use is most probably function calls and imports.

You won't have to actually write the Kubernetes objects anymore, they are generated using helper functions, just like real programming languages would do.

Yes, and as imperial user it is ironic to see the rebels rebuilding the XML infrastructure in their formats.

Apparently the imperial features are there for good reasons.

At least people were pushing back, instead this YAML disaster comes with plenty of hype.

And it's even more brittle...

Kubernetes is very good at keeping track of state and maintaining it. You "just" tell it the state you want and it makes sure it becomes reality (this is the whole point of k8s actually). Terraform is doing the same thing, just for imperative (instead of declarative) APIs (like public clouds, etc).

The issue with Kubernetes is more expressing the state. Kube uses YAML, which quickly becomes verbose and hard to maintain. More on our blogpost: https://grafana.com/blog/2020/01/09/introducing-tanka-our-wa...

Tanka is trying to solve this issue by providing a more powerful language that overcomes these limitations hopefully.

Terraform is really good at describing how infrastructure should be provisioned (VMs, load balancers, dns entries, networking, etc). Provisioning software on a VM and keeping it in a consistent state, however, is not something it's very good at. Userdata is very difficult to do anything complex with (limited size payloads, optimized for uploading a single shell script), and the provisioner system is explicitly described as a "last resort". This makes Terraform not so good at describing how software should be provisioned.

There is a bit of a movement, however, behind using it to deploy software by pairing it with Packer. You use Packer to create an e.g. AMI whose sole job it is to run your software (like a Docker container) then use Terraform to launch a bunch of EC2 instances that have juuuust enough resources to effectively run your software. That'd allow you to eliminate k8s from your stack, though it remains to be seen which stack would be more cost-efficient to run on.

My last company used Terraform to manage Kubernetes. The main issue is that the TF Kubernetes provider supports a limited subset of K8S object types, and of fields within those K8S objects. For example: TF didn't even support Deployment objects until sometime in mid/late 2019 (I may be wrong on timing, but it was long after they were the primary method for general scheduling of long-running containers).

We ended up using TF's Helm provider, sometimes with hacks like a helm chart which deploys an arbitrary YAML file (the so-called "raw" chart). At that point, Terraform is blind to what's actually happening inside K8S. You can still benefit from the ability of TF to pass data from your other infra automation into the Helm charts, of course, but it's really Helm actually managing the configuration of your K8S cluster. And that's the app we all love to hate.

The situation may have been improved, but my conclusion was that it would always be a somewhat incomplete interface.

Oh, too bad :(

The site is hosted on Netlify and seems to be working for most other people.

Maybe reset your browser cache, check your network or try on your phone using cellular data instead?

If the issue persists, I'll take a closer look :D

Delete: No, we need to add that command. For now, use `tk show --dangerous-allow-redirect | kubectl delete -f -`.

Diff: Use `tk diff`. It shows the differences between the local Jsonnet and the cluster. `tk apply` makes them reality afterwards.

Regarding Jsonnet:

1. Documentation is bad. We know that and work on improving that. Some resources that might help:

- https://tanka.dev/jsonnet/overview: Our own docs include some notes about Jsonnet in general for newcomers

- https://jsonnet.org/learning/tutorial.html: Taking the time to read this entire page opens eyes. Annoying and time consuming, I know but worth it.

Regarding Ksonnet and Kubecfg:

1. Ksonnet was magical. Tanka hopes to be Ksonnet without the magic. We got rid of all of those concepts, parameter merging and whatnot. You have Jsonnet and Tanka, a tool that pushes Jsonnet to Kubernetes. That's it. (ok, you also get a lot of handy features like CLI completion, diff and other things to make your dev experience better)

2. Kubecfg is similar, but has a smaller scope. It evaluates Jsonnet and pipes this to kubectl (basically). At the time we started Tanka, kubecfg was by the way part of the deprecated Ksonnet project, so we assumed it dead as well. Luckily it's not, as it is a very cool project, that inspired Tanka a lot.

Tanka after all aims to be like the `go` command: The one command you need to manage your entire complex kubernetes clusters. Also, Tanka is not strictly limited to a single language. For now we focus on Jsonnet, but more may come in the future

Well, Helm did templating.

Tanka does generating, because it seems to be more robust, as the tool understands the output (instead of string substituting a fragile syntax)

See the docs on more details about generating: https://tanka.dev/tutorial/k-lib

Downfalls of a bash based approach: You need to maintain it. And bash is hard to debug, especially when the house is on fire (production outage, etc)

Integrating should be quite straigthforward. Install Tanka, create a new project (tk init), copy your source of truth YAML (without transformations) somewhere under lib/ (for example lib/foo).

Then go to lib/foo/foo, and import each of those yaml files:

   {
     foo: {
       deployment: import "./deployment.yaml",
       service: import "./service.yaml",
     }
   }

In environments/default/main.jsonnet:

  (import "foo/foo.jsonnet") + {
    // patch environment specific things here
    // https://tanka.dev/tutorial/environments#patching
  }

Then use `tk show` to verify it works.

Furthermore, follow the tutorial to get an in depth understanding of Tanka: https://tanka.dev/tutorial/overview