Ask HN: Docker, Kubernetes, Openshift, etc – how do you deploy your products?
I use docker extensively with python backed ansible scripts to manage my product deployments (with a jenkins CI/CD pipeline). That has been a lot of fun, but I have also played with both Kubernetes and Openshift.
I love what Openshift Origin can do, but the learning curve is like a brick wall (See Dwarf Fortress Fun for an example) and the costs are far from minimal.
Kubernetes is easier to learn, but comes with its own gotchas.
What do you do to maintain stable deployments that allow for easy CI/CD? How do you minimize costs with your solution?
115 comments
[ 3.2 ms ] story [ 201 ms ] threadAlso, while I have a vested interest in saying this, you don't always want to solve this yourself. Look at hosted solutions like GCP and CircleCI to make things even more painless.
[0] http://getgandalf.com/
There's still a decent amount of human intelligence involved though, since obviously we want to give customers a good experience. This mainly comes in upfront (where we tune the implementation for each customer) and for any tasks which Gandalf hasn't learned to do yet. I've also invested in making it easier to train Gandalf to do things—for example, I can say "watch me" and then do a bunch of things with the AWS console/API and they get turned into a parametrizable playbook.
Any good DevOps engineer invests heavily in automation. Gandalf is just one level up of automating the process of automation.
If you have any other questions, feel free to email morgante@getgandalf.com.
There's a learning curve, and new features are being added, but at this point I would not hesitate to recommend Kubernetes to just about anyone.
CI: We standardize on CircleCI and it gets the job done, but has some serious shortcomings. I've also come close to building my own on top of the k8s cluster and it's not the correct time investment for me right now, but I'd consider building my own in the future. I've yet to find a CI framework I really like.
I've worked with a self hosted Gitlab instance before and that was nice, but I'm not quite sure how well the Gitlab CI would integrate when the code is on github.
For me, getting GitLab up and running on kubernetes was a breeze (using the popular docker image [1]) and my `pg_dump; duplicity;` backups are chugging right along. I haven't played with their CI yet, but I'm pretty excited to see how much it can do for me automatically managing my cluster.
[1] https://github.com/sameersbn/docker-gitlab
Do you have any advice, or gotchas on getting into kubernetes development?
[0] https://github.com/vyshane/concourse-kubernetes
It is also very nice in that you can execute pipelines, and even just jobs, on the Concourse installation from your local CLI. This makes it much simpler to test pipeline/job tweaks without a single commit being necessary until it is verified to work. Yes, you can do something similar with the Jenkins Pipeline plugin suite using the Replay functionality but it is significantly more tedious (cut and paste from editor into a textbox in the Jenkins UI, bleh).
We switched over from Circle and it's been awesome to scale our build process without fear of resource constraints. It's pretty great :). They even have a CF setup that will auto scale boxes based on CI load.
I moved to Kubernetes about 6 months ago and have been really enjoying it. My first production cluster was hand rolled on AWS, where I found the cloud-provider load balancer integrations extremely helpful (https://kubernetes.io/docs/user-guide/load-balancer/).
I'm now using Google Container Engine which is effectively just a hosted Kubernetes cluster on GCP, which has really been 0 effort setup, and have been deploying to it with Wercker (http://www.wercker.com) [Disclaimer: I currently work at Wercker as of the last few months, but was a fan/user for many years before joining]
One thing I noticed across Openshift, Mesos, and Kubernetes: none of them handle the Docker daemon on a node hanging particularly well, which in my experience happens fairly often.
I'd much prefer doing simple bare-metal deployments again.
I'd take the opposite stance, really. As far as the image format, it's the major differentiation Docker has, and IMO a really clean way of keeping image pulls DRY. Once your hosts have pulled a single image, given that you don't actively undermine it, subsequent pulls, even for different images, only need to retrieve the absolute minimum since they already have hopefully pulled the majority of the file system.
Disclaimer: I am the founder and CEO of Boxfuse
https://kontena.io/
> How do you minimize costs with your solution?
Autoscaling groups triggered off of "cluster capacity".
I have github hooked up to travis. When a new PR (or commit) is pushed travis shoves the app into its container, and runs the test suite inside the container.
If that passes AND the branch is master we push the image to docker hub. As of now we manually update the app inside of rancher but I think automating that will be a simple API call. Once we get more stable I'll be investigating that.
I still haven't quite figured out secret management but outside of that and a tiny learning curve it's been pretty smooth sailing.
An example travis config: https://github.com/OperationCode/operationcode_bot/blob/mast...
Automating the upgrades (i.e. redeploys) in Rancher is pretty straight forward - their API is super easy to use. I ended up writing a simple tool in mostly Bash to handle it, and threw it in a Docker container to run on Teamcity.
They also have CI/CD out of the box and builds can be triggered in your existing cluster with a 'convox build' or triggered on pushes to your private github repos.
Overall, unless you have a team that actually sees benefit in managing your own container and cluster manager(you better be big), id recommend embracing Convox, or something like it. The complexity still exposed by Kubernetes, OpenStack or ECS is still significant.
A nice bonus is that they only charge you for the minion nodes. The Kube master is free.
Disclosure: I work for Pivotal on Cloud Foundry, a competing system.
https://cloud.google.com/container-engine/pricing
(I work for Google Cloud)
I'm one of two developers at a very young startup, and the one responsible for backend + devops stuff. I simply don't have the time to learn a more complex tool like Kubernetes (not that I didn't consider it) while also working on the actual product. Its simplicity has been a bit limiting on occasion, but they're happy to accept PRs for well thought out changes. I recently had a PR merged regarding UDP ports and ELBs that should make microservice architectures much easier and cheaper to implement.
https://cloudcraft.co/view/5582ddd4-c6f8-4354-8f5b-9fb0a3744...
* Development: docker + docker-compose. Ideally, we would want to get rid of docker-compose for development.
* CI: Travis (planning on switching to something that is more on the CD side)
* Infrastructure management: terraform
* Prod: AWS, CoreOs, Kubernetes 1 master node and 5-6 worker nodes (m4.large) in an autoscaling group.
Infrastructure deployments and updates are done by Terraform. Blue/Green deployments thanks to the autoscaling group.
Kubernetes deployments and updates are done by kubectl.
There's still problems with each piece, but for the most part they work great without much trouble.
I, personally, prefer the bare-metal deploys of automated scripts. I usually just spin up a VM and write a bash script to "prep" it the way I want. After that, I just run "./deploy" and it pushes where I want. I like this because I feel like I have more control and it actually feels easier. Plus, I've run into weird issues with Docker that take so long to debug that it completely cancels out the benefit of using it for me.
The bash script I have works for every side project I create, and is simply copied from project to project. :)
You're mixing terms here. A VM is not bare-metal.
I meant just an old-school deploy without containers and the sort.
I use Gitlab CI and helm[1] for deploying. The last step of the ci process checks out the helm chart which is just another git repo and executes a helm install/upgrade CHART-NAME. Making things accessible is done through kubernetes ingress with nginx[2](which includes getting let's encrypt automatically for all external endpoints) so when I want to deploy a new staging version of the app I can do helm install app --set host=my-stage.domain.com .
There still a few gotchas like the pods won't update when a configmap was changed which is important because I keep the container configuration maps as configmaps. A crude workarround for this is [3] which triggers a configuration reload of the application running inside the container.
This solution has no licensing cost unlike Openshift(Tectonic[4] is another enterprise Kubernetes distribution which is free for 10 nodes) and the cost are based on the amount of time to set this up. But after you got into helm and more complex kubernetes deployments it should be easy.
[1] https://github.com/kubernetes/helm
[2] https://github.com/jetstack/kube-lego
[3] https://github.com/jimmidyson/configmap-reload
[4] https://coreos.com/tectonic/
[1]: https://github.com/Eneco/landscaper
thanks for the info. As far as I understand it you configure the state of the helm release + configured values in a file and apply it to a kube cluster. Depending on your cluster setup this is really helpful. Do you have a solution for triggering a rolling update of pods if a configmap has changed? In your examples I didn't see any configmaps.
Our current approach is using an Operable (https://operable.io) Cog we wrote which takes the kubernetes yaml and applies it to a running cluster. It's not perfect, but I'm pretty happy with the direction it's going. We built this cog in a public repo (https://github.com/retracedhq/k8s-cog) so you are welcome to use any of it, if it's useful. Then we have our CI service send a message (using SQS) after a build is done to deploy to staging.
Lately I've been using home-brew chatops to manage products so it's nice to hear what other people are using. Operable looks really interesting, I'm going to give it a shot. Thanks for the example cog!
http://rancher.com/
Rancher was the first thing I'd seen that claimed to be able to pull it off. I'll definitely give this more of a look.
Rancher is also a breeze to deploy, I could manually deploy it with one hand tied behind my back in 10mn on AWS.
We deploy all our containerized applications to Rancher (using Cattle for orchestration) via Jenkins jobs with a standardized Makefile for build, test, and deploy, making things consistent.
We look at running straight k8s, but it was like using a chainsaw to sharpen a pencil for our use case.
In addition their devs are extremely helpful and also have a hobby of getting things to run on ARM.
The biggest challenge right now is the ingress/loadbalancer abstraction. Hopefully, that should get resolved over the next few months.
I also attended the DevConf.cz and saw a lot of presentations regarding Openshift. They have most of the talks on youtube (https://www.youtube.com/channel/UCmYAQDZIQGm_kPvemBc_qwg) in case somebody is interested
I was leveraging EFS as NFS mounts for my persistent volumes and had good results.
You might check out fabric8 if only for their visualizations of what is going on in your openshift / kubernetes environment.
Thanks for the youtube link! I'll be sure to check it out
https://github.com/redhat-openstack/openshift-on-openstack/
Back in 2015, I implemented a Kubernetes by hand in AWS. I'm not going to do something like that again. GKE is fairly painless and it has most of the sensible defaults that I want. Networking just works -- pods can talk to each other as well as to any VM instances from any availability zone and region. Integrating with GCP service accounts just works. Spinning up experimental clusters is easy, as is horizontally scaling the clusters. One gotcha is that Google has not made K8S 1.5 generally available in all regions or availability zones. Otherwise, upgrades are pretty easy.
I have deployed with Docker Compose (not doing that again -- it is easier to use shell scripts). I have deployed with AWS ECS service (not doing that again; it does not have the concept of pods which severely constrains how you deploy). I used to deploy with Chef. I've heard of Chef's Habitat, but have not played with it.
Back for the 2015 project, I wrote Matsuri as a framework to manage the different Kubernetes templates. It's useful if you know Ruby. It uses idiomatic Ruby to generate and manage K8S specifications, and run kubectl commands. I wanted a single tool that could work with all the different environments (production, staging, etc.) as well as manage the dev environment. For example, if I want to diff my version-controlled spec on dev with what Kubernetes master currently has, I would use `bin/dev diff pod myapp`. If I want to diff the deployment resource by the same name, I would use `bin/production diff deployment myapp`. I can write hooks specific to the app. For example, `bin/production console mongodb` uses hooks to query Kubernetes to find a pod to attach to, determine the current Mongodb master, and invoke the command to go directly into the Mongodb shell. But I could have invoked `bin/staging console mongodb` or `bin/dev console mongodb`. I could do this because I have been developing software for a long time and I have enough ops experience to be able to put it all together. YMMV.
We're using Go.cd for the CD. I could have used Jenkins, but decided to give Go.cd a try. Go.cd has some advantages (such as much better topologies and tracking value streams) though there are also things it does not do as well as Jenkins (Go.cd auth mechanisms blow, and I had to write my own custom proxy to get Github hooks working more securely and reliably). Setting up GCP service accounts so that go.cd agents can deploy was a lot easier than I thought, once I read through the GCP docs. (Much easier than AWS).
Docker containers are still difficult to make. You want to vet things before using them. Handling this stuff is still going to be a full-time job for someone, both in terms of designing the infrastructure as well as the development tools. There are a lot of issues that come up because dev might throw things over the wall that might impact the overall reliability and performance of the system.
What have you found are the biggest advantages of pods over containers? How does ECS constrain how you deploy? Are you simply referring to rollout/rollback, scale up/down?
That meant I could not horizontally scale one container more than the other. I can scale the whole group, but there is a lot of wasted resources at that point.
Kubernetes pods group containers together under a single IP address. Containers from one pod (one IP address) can talk to any other pod. Docker did not even have this functionality until 1.12, and that is too little, too late. (And I am not sure this is something ECS supports right now). Combined with label selectors, long-running services (which binds a DNS name to the set selected by the label selectors), I can horizontally-scale pods and still maintain service discoverability. Using DNS makes service discovery stupid-easy. This means I can scale Kubernetes pods independently from each other.
Another consequence of using Service objects to select a set based on label selectors is routing can now be dynamic. Pods that need to talk to another pod goes through the service. I can then scale the dependency up and down, and it doesn't really affect the pod that requires that service. I can do rolling upgrades to the dependency, and it works because Service abstracts that through label selectors.
There are still some warts related to this setup. Stateful sets still needs a lot of work. I've also found that many applications caches IP addresses (redis sentinel being a notorious example). To work well with Kubernetes, it's better to always query DNS when making a connection. Ruby drivers for Mongodb and Redis, for example, will cache DNS lookup, making failover fragile (if you are running Mongodb and Redis inside Kubernetes; if you're not, you won't have this problem).
I was choosing between Kubernetes and Mesos after ECS, but had not looked into either deeply. It was random chance that took me to Kubernetes instead of Mesos. Kubernetes solved many of the pain points of Docker Compose and ECS.
It's largely undocumented. I've heard some interest in places to use it. Time constraints is such that creating examples for it is low on my priority list. Frankly, if you don't know Ruby, you're probably better off looking at Helm.
Do you have large workload spikes, or traffic spikes?
The build jobs creates images that are published to ECS repositories, and there are auto scaling groups that add and remove engine hosts to and from ALB target groups for each deployed service. It makes service discovery, scaling, etc. really easy.
Definitely try swarm out if you haven't already. 1.12 was good, 1.13 is amazing (secrets, health-based VIP membership, etc).
we have been considering playing with swarm. How many images and instances do you have, etc
And especially, how have you leveraged 1.13
Example kube-deploy files: https://github.com/UKHomeOffice/kube-piwik
Example app / drone files: https://github.com/UKHomeOffice/docker-piwik
Platform Documentation: https://github.com/UKHomeOffice/hosting-platform
KD - our deployment tool https://github.com/UKHomeOffice/kd
I can't really comment on whether or not this specific pipeline actually works as I've just picked a random open source example but the workflow is there.
We also have a legacy tool and use jenkins sometimes, but mostly that won't be open sourced.
Legacy deployment tool - don't use this. https://github.com/UKHomeOffice/kb8or
This is Kubernetes, plus monitoring of your choice, running on your infrastructure, remotely managed by our team. The side benefit is that the same setup works on different infrastructure options, so you deploy and run the same stack on AWS and also on-premise/bare metal.
Jenkins has a plugin that integrates with elastic beanstalk. This makes ci/cd straightforward.
There's no extra cost for elastic beanstalk, other than what you'd pay for ec2, s3 and elastic load balancer.
We've a starter template with a bunch of .ebextensions scripts that simplify common installation tasks.
If your application is a run-of-the-mill web app speaking to a database - elastic beanstalk is pretty much all you need.