Ask HN: How do you go about starting to develop an application for Kubernetes?
I'm curious, what's the typical lifecycle of a developer in the space of K8s?
Imagine this scenario: Your manager said "hey we are taking a bet on Kubernetes, now go figure out how to run our apps there."
Do you:
1) Start by setting up local dev environment, use docker locally, build manifest/helm files and test them in a local minikube, and finally think about now moving your code from local dev machine to a cloud managed K8s provider? (AKS, EKS, GKE, etc)
or
2) Go create a cluster first on the cloud (AKS, GKE, EKS), and then learn that you need a docker file, manifest files, etc from the portal and then use the portal as a way to help develop your app?
or 3) Something else?
9 comments
[ 2.1 ms ] story [ 15.4 ms ] thread1. Write a dockerfile for the service and test if I can run the service locally in docker.
2. Push the dockerfile to AWS.
3. Cobble together Deployment, Service and Ingress yaml files from reading tutorials and docs, then apply those to the cluster and observe what happens. Repeat until it works.
4. We slowly iterated our tooling and yamls as we learned more.
By and large, your option (1) is closer to what we did, but with the caveat that most of us didn't stay on the "use Minikube" path very long, because it's somewhat limited compared to a real cluster, and can really bog down a developer laptop, etc.
Also, consider that we were coming from a situation where our deployments were to Docker Swarm, so we already had all our apps built for containerization in the general sense, and just had to port things from Docker Swarm to Kubernetes. Not quite the same as starting from scratch.
Anyway, we got automation in place for provisioning AKS clusters and made it easy for developers to quickly spin up a small (2-3 node, with auto-scaling optionally turned on) cluster on AKS and the ability to directly interact with the cluster using kubectl. Most developers quickly started doing all their initial prototyping and testing on a real cluster instead of Minikube.
Early on we hand-wrote K8S deployment yaml files and ran deployments using kubectl apply -f filename.yml. But we quickly moved to Helm and by and large I'd say that was a good move. Most deployments now are doing using Helm, run under an Azure Devops pipeline (but you can still do all this stuff on the local command line if you want, at least in non-prod).
The good news is, you can do a lot of this in something of an incremental fashion. That is, you don't have to use every feature of K8S and every fancy supporting tool right out of the gate. You can start by taking as basic Docker image and just getting it to run and configuring a NodePort for access. Over time you can start integrating Secrets, ConfigMaps, Services as External LoadBalancers, persistent Volumes, Helm, yadda, yadda, yadda.
We also adopted Istio as part of our stack early on, and that has also been a big success for us. Istio provides a lot of facilities that are very handy and/or desirable and has been very helpful for us. The biggest win is the mtls support which makes it easy for us to comply with the (internal) requirement that all "on the wire" traffic be encrypted. But beyond that, the traffic management facilities with VirtualServices have proven quite useful. We're just starting to explore integrating Istio's traffic management with Azure Active Directory using OIDC now. If it works, that's also going to be a big win for us.
Note that we did have to make some code changes, since our old model used the Netflix OSS stack along with Spring Cloud Config, and once we went to K8S we were able to throw out Eureka, Hystrix, and some of that stuff. Spring Boot does have a handy library that provides "native" interop with K8S via the API server to automatically import ConfigMaps and what-not, which is kind of handy if you're a Java/Spring shop.
Last note: me personally, I don't find the Azure Portal to be something I use very much, but I'm biased towards being a command-line guy in the general sense. Most of my interactions with Azure and AKS are through the az and kubectl command-line tools (and occasionally istioctl or helm as well).
1) When you say built automation to spin up clusters in AKS, what did you use and what entry point you used to spin up those cluster? E.g: Did you have a terraform template that was used through cli?
2) How were the Helm files deployed? Did you have a CD pipeline that deployed helm on every code commit?
3) When you said iterating on the app directly on AKS, what was the iteration? Is the scope, you had an existing app and you were iteratively porting the code base?
On a meta level, how many of your experience you and your team preferred a UI/UX experience for? For creating helm charts, did you feel like a portal experience would have made it easier or did existing documentation work for you? What other dev cycle you and your team wanted guidance from AKS portal?
It's pretty much all ARM templates and AzDO pipelines. No Terraform. We're only just now starting to go down the path of evaluating Terraform to become part of our automation stack.
2) How were the Helm files deployed? Did you have a CD pipeline that deployed helm on every code commit?
Git repos in AzDO and AzDO pipelines. I don't know if any teams are deploying with every single commit or not, but there's definitely a set of AzDO pipelines for building the code, pushing the images to DTR (or ACR) repos, and running the Helm deployment.
3) When you said iterating on the app directly on AKS, what was the iteration? Is the scope, you had an existing app and you were iteratively porting the code base?
I don't know how to answer that. It was a very ad-hoc, "figuring it out as we go" kind of thing.
On a meta level, how many of your experience you and your team preferred a UI/UX experience for?
I'm not sure. I know there are people who do like going into the portal and doing stuff, but it's not my cup of tea. And my role was on a team that provided supporting services in a kind of "cross-cutting concerns" sense, not a team that owned a specific user-facing application. So I don't have complete visibility into how all the various AppDev teams chose to do certain things.
For creating helm charts, did you feel like a portal experience would have made it easier or did existing documentation work for you?
For us, we basically had a small group of people who developed the first series of Helm charts, and published them. Once that existed, I think a lot of other teams just took those artifacts and used them as a template and modified them to suit their project.