HN Explorer

I rewrote in Crystal my tool to quickly create Kubernetes clusters in Hetzner

127 points by SkyLinx ↗ HN
Hi all, I just released a new version of https://github.com/vitobotta/hetzner-k3s that is a complete rewrite in Crystal that is faster, better overall and allows me to offer simple standalone executables. The previous version was written in Ruby, so users had to either set up a Ruby environment, or use a Docker image instead. This is no longer necessary.

This is my first project in Crystal and I love it already! It's syntax is lovely - similar to Ruby's - and it's FAST!

I would appreciate any feedback anyone might have - if you are a Kubernetes user as well as a Hetzner customer, please give it a try and let me know what you think, if you encounter any issues etc.

During my tests I was able to create a cluster with highly available control plane (3 masters behind a load balancer) and 5 nodes in multiple regions in just *3 minutes*. This is the total time to create the resources (servers, firewall, load balancer, private network), deploy Kubernetes to all the nodes, install some stuff required to provision load balancers and persistent volumes out of the box, and install an upgrade controller to easily upgrade the Kubernetes version (I use k3s as Kubernetes distro). I don't think there is a faster way to create a production Kubernetes cluster!

Do you have Kubernetes clusters in Hetzner already? If yes, which tool did you use to create them? Would be interested to know more about alternatives people are using.

48 comments

[ 2.8 ms ] story [ 94.7 ms ] thread
xrendan
I have a cluster using terraform-hcloud-kube-hetzner [0] and I'm quite happy with it.

It uses MicroOS for rolling releases and auto-upgrades for less maintenance. I'm quite happy with it so far, and I like that it integrates with terraform.

[0] https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...

SkyLinx
I had never heard of MicroOS before, sounds interesting. Before building this tool I actually used both Terraform and Pulumi. My tool is a ton faster setting things up and only requires a small YAML config file. Would be able to give it a try? I am looking for feedback :)
stephankoelle
You might then be interested in Fedora CoreOS, we use it with Hetzner cloud and real Hetzner root server with good success. (Also with ovh)
SkyLinx
I can test it. What would be the advantage if I use it in my tool instead of Ubuntu? At the end of the day with Kubernetes I don't think it makes much difference which OS you use. I picked Ubuntu because it's standard and servers with this image build very quickly with Hetzner Cloud.
mdaniel
CoreOS (and its spiritual successor Flatcar) are immutable, which gravely limits the silliness that can be done to them post-launch. We use now use Bottlerocket since we're on EKS, but it's even more locked down since to even get an interactive shell is some major hoopjumpery. I believe Talos goes even further and is completely devoid of a shell

So, yes, it absolutely matters which OS you use, of course depending on your threat model and tolerance for "no, you don't get to ssh onto a Node and do whateverthehell you want"

evrflx
How did you like pulumi? Why did you use both terraform and pulumi at the same time, my understanding was that they serve the same purpose.
SkyLinx
I used Terraform first and then switched to Pulumi. What I like of Pulumi is that you get to use a proper programming language rather than a DSL, so you have more control/power in a way.
wise0wl
We use Pulumi at my current workplace and love it. Being able to use a real language (Typescript, in our case) allows us to create data structure abstractions to feed our Infrastructure-as-Code that make sense from a developer interaction point-of-view. Our abstractions are less leaky and more in-line with a well designed interface, ready for use by a team that doesn't need to know all the details. We can have developers write feature code and design services and just plug in simple details for new services in our stacks Yaml.

However there is a gotcha. There are nearly infinite ways to setup your Pulumi codebase and there is no real prescribed structure outside of "create stacks". This can lead to a lot of time spent in architecting / implementing / testing different structures and abstractions which are not really a thing with Terraform. This can take time and can lead to refactors if you are not extremely intentional with your design.

incompleteCode
How much does this cost you on a monthly basis?
SkyLinx
I can give you an example of one cluster than I use for personal stuff:

- 3 masters with 3 cores and 4 GB of RAM each - load balancer for the masters (for the Kubernetes API) - 3 workers with faster AMD CPUs and 4 cores/8GB of RAM - around 200GB of storage in persistent volumes

I think it costs around 60-70 euros per month total! I can't remember the exact amount because I have many things with Hetzner and do tests creating and deleting stuff all the time so my bill is rather long lol.

Hetzner is very cheap, but I can voice for the performance and reliability as well. It's a very good provider and I highly recommend them. Now they even have 2 locations in the USA!

dakiol
Nice price. Are those dedicated servers? Or just "cloud" (standard or dedicated vcpu?)?
SkyLinx
Cloud servers with standard shared cores. They perform really well
incompleteCode
That’s cost-effective! I’ve been toying with the idea of running a personal distributed system that’s not serverless to just hack around and Hetzner seems approachable going by your experience.
SkyLinx
Definitely. I have been using them for years and I still can't understand how they can offer such a good service at their prices, considering how much the competition charges.
unity1001
Is your control plane high available? How hands off is it?
SkyLinx
It's up to you. You can configure it with one master or HA multiple masters (e.g.). If you set the instance count for the masters pool >1 then my tool will create a load balancer in front of the API server.

The tool simply requires a small YAML config file and creates and configures everything that you need to get a fully functional cluster in a few minutes. It creates all the infrastructure resources (servers etc), it deploys kubernetes on all the nodes, and then installs some software that allows you to provision both load balancers and persistent volumes. It also installs a controller that makes upgrading to a new version easier.

What the tool does not install is other apps, because this would be too opinionated and depends from cluster to cluster.

mathverse
Love it! Have you tried to create a k8s operator in Crystal?
SkyLinx
Not yet, but I am definitely going to use the language more! For example I want to rewrite a controller I wrote in Ruby that monitors Velero backups and sends me notifications via emails, Slack, or webhooks so I can know when a backup failed. I did this in Ruby but want to rewrite that too.

In terms of operators I have an idea for an operator but it would be very specific to our infrastructure and backend apps at work.

eurg
Hetzner has some attractively priced dedicated servers - for some heavier workloads that might be nice. Do they use a very different management API? (They have a cancellation period of "30 days to the end of the month", which would complicate matters.)
SkyLinx
I don't think there is an API for the dedicated servers AFAIK, only cloud.
[deleted]
(comment deleted)
hbien
Great job! How were you able to do a stand-alone binary for Mac? I was able to for Linux via their Alpine/MUSL image, but never figured out how to for Mac.
SkyLinx
Just using Crystal as installed with Homebrew and then running `crystal build --release`. Very simple, see the Github Actions workflow in the repository :)
hbien
ah, are these are "standalone" or static binaries with no dependencies then? I think end users may need to have the crystal runtime installed as a dependency.
SkyLinx
At the moment they are not static, so on Linux you may need to install a couple of packages depending on your system. I am going to try and compile a static version. On MacOS AFAIK it is not possible to compile a static binary.
SOLAR_FIELDS
Just as a point of reference, for me to spin up a 3 node cluster in terraform with GKE it takes 10+ minutes.
SkyLinx
Yep. I tested creating a cluster with 5 nodes and it took almost 20 minutes for me with GKE (just from the console, not Terraform).
MuffinFlavored
Weird question: you use Terraform to manage the cluster nodes themselves, and then you use k8s YAML (or Helm charts or something else) to manage the k8s resources, right? You don't manage the k8s resources from Terraform, do you?
SkyLinx
Correct. I can't remember the details because I haven't checked this stuff recently but that's the gist of it
SOLAR_FIELDS
Correct. I like to think of it as “infrastructure” (things managed by Terraform that define the structure) vs “deployments” (actual apps deployed on that infrastructure). The line gets blurry because you can actually deploy many types of apps with Terraform too if you want, but typically you would use Helm or similar solution for that.
gerwim
Looks great! I see your tool uses Hetzner block volumes for storage. How do you manage your data backups (as Hetzner does not offer anything for their block storage volumes as far as I know).

My current setup is also on Hetzner. 1 master and 3 worker nodes. Managed by Rancher and I use Longhorn (also from Rancher Labs) for storage. It runs stable for two years now. But looking for a simpler solution, so I looked into k3s and k0s last week -- will also look into your tool ;-).

SkyLinx
Give it a try, looking forward to hearing what you think :)

For backups I just use Velero. It can back up any kind of volumes with the Restic integration, and it has been rock solid for years for me.

In the past I've used Longhorn, Rook, OpenEBS and others but really, in Hetzner Cloud just use their volumes. It's much more reliable than those software solutions in my experience and performance is also a lot better.

scary-size
Sweet! I have no interest in Kubernetes, but this lead me to give Crystal a try. Feels solid so far, nice standard lib for the usual things (json, http etc.). Compiler warnings helped find errors quickly. The language documentation reads very easily. And best of all: standalone executables.
boredtofears
I used it a lot over the summer on some pet projects and really enjoyed it. Being a heavy Ruby/Typescript user in recent years it feels like a natural melding of the two. Macros are a really interesting way to achieve many of the same metaprogramming properties of Ruby (although a little awkward to reason about).

I used Jamis Buck's book The Ray Tracer Challenge[1] as a fun kind of way to learn the language and ended up learning a good chunk of the language as a result.

My biggest complaint is just that it's not more popular - it's ecosystem is pretty web developer-centric and I'd personally love to see it used for more things outside of that.

[1] https://pragprog.com/titles/jbtracer/the-ray-tracer-challeng...

SkyLinx
That page doesn't mention Crystal. Is the book about Crystal code?
boredtofears
No, it’s about building a ray tracer. You can use any language you want to work through the book. I find it a good way to explore new languages.
SkyLinx
The standalone executables were the primary reason why I looked to rewrite the tool in something other than Ruby. I actually tried to use a sort of "compiler" for Ruby but the executables were huge, took long to build and had issues.
sureglymop
If I'm allowed to ask.. what's Crystal?
SkyLinx
It's a programming language that looks very similar to Ruby in syntax (if you are familiar with Ruby), but it's compiled, so by switching from Ruby to Crystal for this tool I was able to provide standalone executables which are easier to use. With the Ruby version instead my users had to set up Ruby or Docker, which was not optimal.
Alifatisk
Regarsing the standalone executable, do I only have to compile it once to be able to run on Win, Linux & Mac? Or do I have to compile it on each OS?
SkyLinx
You need to compile a different binary for each OS as they are not compatible :)
wg0
Cannot stop myself from commenting because it has everything of my interest. Kubernrtes, Hetzner and Crystal.

Another kubernetes installer. I think there's one Kubematic or similar already for Hetzner.

I'm going to try this and be possibly contribute back.

Few questions:

Why k3s why not k0s? What about cluster upgrades? Are they covered?

I'm probably going to setup a staging environment but can anyone suggest me a good CI/CD for Kubernetes? How about GitHub actions?

Secondly, I need to run databases lots of them some in range of 500 GB or so and backup/restore is important. Any suggestions there?

SkyLinx
k3s because it has been around for a while, and it's something I have used almost since it was first made available.

Yes cluster upgrades are covered with the "upgrade" command, as described in the REAME. I use the Rancher's System Upgrade Controller for this, which makes rolling upgrades very easy and super quick.

Github Actions is perfectly fine, it's what I use both for personal stuff and at work.

For the database, are you planning to host it in Kubernetes? Depending on the database type there may be an operator that can handle backups easily for you. For example for Postgres I have used the Zalando operator which does this and there is a newer one called CloudNativePG which might be even better. What kind of databases are they?

speedgoose
I have a development kubernetes cluster on hetzner. I used Ansible and K3S.

The most challenging part to me is setting up the disk encryption using LUKS, it requires a few manual steps.