Has anyone already successfully created a HA-cluster on a bare metal infrastructure?
It seems overly complicated and not that well documented from what I can see.
I have attempted this using https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare... . One can get quite far, but I am not sure it is worth the effort. I understand this is a vague analogy, but it feels like you're trying to setup email infrastructure when what you really want is to send email.
It is a lot easier on the public cloud, or easier still on a managed service.
What about people who cannot go to the cloud? What about people who need more performance? Lower costs? Kubernetes seems to be such a great abstraction for the underlying hardware, why not use it where it is needed the most?
People do run Kubernetes on bare metal... but there's no denying it takes effort. That effort should be coming down over time, but unfortunately there's still a ways to go.
That's not entirely true. I attend meet-ups with engineers from a fairly large company whose name you would recognize who run K8s on bare metal. I know that's vague, but I don't know how public their infrastructure is, so I'd rather not speak out of turn.
You're absolutely right that it takes some effort.
Bloomberg runs Kubernetes on bare metal and one of my friends is an engineer on the project (but did not speak at KubeCon about it). They've got a relatively sizable install as well.
citation required, since there are IaaS offerings that do not involve a hypervisor (e.g. Rackspace OnMetal) and therefore its possible to move "to the cloud" without performance penalties.
on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
> and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same.
And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo.
For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
citation required for what? For stating, that it is complicated to install on bare-metal and that it's been a priority for Kubernetes to be deploying on cloud servers?
I felt very similar. Especially thinking about it from a disaster recovery/re-bootstrapping perspective. That scared me into using docker swarm kit which is stupidly easy to bootstrap. Keep in mind that's not necessarily a like for like replacement but it was enough for my use case.
I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
What would your ideal setup look like on baremetal?
The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months.
I've deployed kubernetes on aws as if it were bare metal. Try to run through https://github.com/kelseyhightower/kubernetes-the-hard-way translating what they do there to your particular setup(configure firewall, add routes to router, etc)
I'd suggest looking into kops (https://github.com/kubernetes/kops) for Kubernetes on AWS, whether you let it take the actions or you tweak/use the Terraform config it can generate
+1 there has been a discussion of making a sig-on-premise/sig-bare-metal (with a fair bit of opposition from others) [1]. Would help to have your comments there as well.
The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset?
How do you set up ingresses to pass through source ip.
a lot of this is taken care for the cloud...but NOT for bare metal.
Stackpoint user here; running a very small (3 node) cluster on DigitalOcean currently using their service. It's been relatively painless so far, but I'm pretty new to k8s so my needs are very basic.
In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
There will be a 1.5 compatible release coming soon.
Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
Tons of great suggestions in the thread, but let me also point you to the kubeadm work - in which in Kubernetes 1.6, folks are trying to make it easier to install HA setups.
Folks wanting an easy road to deploying Kube on (so far) AWS, GCE and OpenStack, with bare metal and local on the roadmap, might want to check out KubeNow:
Tries to re-use as much as possible from great projects like Terraform, Packer, Ansible and the kubeadm tool, and just add a thin layer on top of that (less risk for bit rot), which is an approach that seems appealing to me.
Isn't IPv6 mostly for internet traffic, where 32 bits of address space isn't enough? Why not just route IP6 traffic to a Kubernetes cluster, and have that cluster use IPv4 internally? Is there something more to IPv6 than the increased address space that I'm missing?
Then you need gateways that keep track of and translate every connection, and special ways to tell your backends the client IP (e.g. for logging, fraud detection, ...). Dealing with IPv4 to clients with an IPv6 backend is easier than the other way around.
This works on GCE, AWS, Azure, VMWare, bare metal (via MAAS) and LXD containers. We're currently supporting 1.4 but will support 1.5 (and upgrades) in the next week or so.
As Marco alluded, our only supported storage mechanism to date (Which is represented and managed via the charm) is Ceph backed by RBD storage.
We have plans on including other storage vendors and mechanisms, but they aren't on the roadmap for the very near term. If end users start requesting a specific storage solution, it would go in our planning doc and get added to the roadmap. We're quite active with our early adopters that give us feedback and file bugs/requests.
To date you could continue to use alternative storage providers such as NFS or Gluster - but we don't have the PV creation+enlistment captured in the charm code just yet, but again, due to priorities. End users pretty much set the priorities for us, and we then circle back with some light weight planning and execution.
What does Canonical work on in upstream Kubernetes? I see hundreds+ of commits from Redhat and CoreOS in Kubernetes and the associated projects?
In the main kubernetes git repo, I see 39 non-merge from @canonical commits, 99% of which are under cluster/juju/* and 1 doc fix for juju, so not really a value add to Kubernetes at all. How can I, as someone deploying kubernetes on bare metal, trust you to manage a difficult project when Canonical doesn't seem to contribute much at all to the project or ecosystem?
It is great that you're making kubernetes easy to manage and deploy, but other than ease of use (which GKE does an excellent job of as well from the authors of k8s), what is Canonical's actual value add for paying you to manage k8s? Sorry for the terseness, but I'm genuinely curious what value add there is here. As a purely tech focused person, I simply don't see it. Even other firms doing exactly what you do (Apcera) have actual code change commits. I don't see a single one from Canonical that does.
In defense of Canonical, it's preferable that an organization leave the code alone rather than influence it for their own benefit. If Canonical is able to offer a useful embellishment upon k8s without changing much k8s code, good for them. (I have no stake in this personally, though I am still a bit spiteful toward Canonical over Unity :p )
Unlike other companies, we don't jump in to upstream projects and throw weight around to influence a project to make it more marketable for us. Kubernetes has a vibrant, and powerful core contributor base already. When it makes sense, for any changes we make, we'll be happy to package those back upstream. Primarily, our contributions to Kubernetes is around operations. What happens 1 week, 1 month, 1 year after setting up a cluster.
GKE is great, but GKE is Google only. It's not on-prem, it's not cross cloud, and it's not portable. That's important to some people. Our contributions to cluster/juju is the distillation of our operational knowledge in running Kubernetes everywhere. The same upstream k8s, deployed with the same tooling, everywhere.
The way you start this off is a pretty poor response to my original question, whereas the rest is fantastic. It isn't about throwing one's weight around or even making it more marketable, but about improving the product for new use cases (such as Apprenda wanting to better Kubernetes via the on-premise sig).
Thanks for your response, I was curious as to the value add and this helps.
Thanks for the feedback, my original sentence may have been a bit hasty, but the core of the message is there. I see Canonical's role as the expediter/server in a kitchen instead of donning another chef hat when dish are piling up to go out. We want to celebrate and get the amazing work of the community into as many hands as possible.
We participate in SIGs as well, sig-on-premise being one we co-chair and co-founded. We've planning on helping the project in ways other than code contributions.
Sorry I meant even Apprenda has actual golang feature commits. That was my point Re: Apprenda and contributions.
This seems to be Canonical's general direction. It just makes me sad that is is more on marketing than engineering. I wish it was more of both as clearly Canonical does Linux + marketing better than basically anyone or there wouldn't be so much Ubuntu everywhere :)
Operational ease of use and distribution is our contribution, and we've only just recently joined the Kubernetes community, but as pointed out we see our role value as getting Kubernetes into as many hands as easy and quickly as possible.
EDIT: My first response might of come off as rude, so I fixed it.
That's fair, your response seems reasonable, and Canonical has done an excellent amount of this. Setting up kubernetes is hard, but should be surmountable by anyone wanting to seriously use it. Fixing it when it breaks, or figuring out how to run apps when there are issues is what is hard. Perhaps I rely too much on core engineers to fix those issues when it happens and discounted the value add proposition of Canonical on this.
Looking at commits to the code base alone is a risky way to judge engagement. It would be like saying you cannot use AWS unless you contribute to the code. We saw the same false logic in the OpenStack community that drive a lot of marginal activity into the code base instead of demanding operability.
We really want these projects to have vibrant ecosystems. Canonical (and others) is contributing to that and helping users build solutions based on the existing code base. Adding features is great. Adding users is equally important.
You certainly could. You'd be incurring another service running on azure to act as the disk broker. A lot of end users prefer to use things like the providers persistent disk and acknowledge the limitations there in (like 16 disk maximum per instance, and heightened costs for having managed storage)
But there's nothing stopping you from enlisting Azure PV's as a resource, Ceph managed PV's, and other incantations of durable storage. I only ask that you really consider the cost/benefit of each, and pick what makes the most sense to you.
My thoughts would be to use the azure PV disk type, and if that's not dynamic enough to meet your needs, then enlist ceph + large volumes and carve those up into RDB's to share among your workloads.
I'm sure there are others with differing opinions, and I'm happy to help you work through them (but not on a HN comment thread)
Seek me out in Slack as @lazypower, or ping me on the Juju IRC channel irc.freenode.net #juju I'm @lazypower there as well.
And finally, our juju user mailing list is another great resource for supporting questions like the above:
k8s-the-hard-way explicitly says its not production ready. Is there a document that explains what its missing or production topology documented somewhere?
I don't think its documented in any official capacity, but we (sig cluster ops) did generate some visuals that might aid in grokking the topology of Kubernetes as a whole, and we did model this after production-setups.
A few things to keep in mind:
These maps are service centric, and abstract units as vertical columns in their respective diagrams. Services must be HA to be considered “production ready”
Additional concerns that may/may-not be represented here:
- TLS Security on all endpoints
- TLS Key Rotation in the event of compromise/upgrade/expiration
- Durable storage backed workloads
- ETCD state snapshots for cluster point-in-time recovery
- User/RBAC - this still needs more info before i can outline it (time limited)
- Network policy for namespace/application isolation (this is an unspoken requirement for many business units)
We left off working on a Network draft diagram, and if you’re interested in contributing/participating in this process, join us in the #sig-cluster-ops slack channel. We meet thursdays (or have, new year schedule dependent)
One note of caution for those skimming the release notes: v1.5.0 now allows anonymous access to the API server by default. With the default k8s authorization being `AllowAll` this makes it very important to pass `--allow-anonymous=false` if updating from v1.4.x to v1.5.0.
66 comments
[ 4.7 ms ] story [ 133 ms ] threadIt is a lot easier on the public cloud, or easier still on a managed service.
You're absolutely right that it takes some effort.
on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same.
And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo.
For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months.
[1] https://github.com/kubernetes-incubator/bootkube
[2] http://kubernetes.io/docs/getting-started-guides/kubeadm/
Look at Openshift, ansible playbook installer for Fedora 24. You can configure 2 masters/ha/etc in the playbook.
Disclaimer: I'm on the team that works on this.
The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23
https://github.com/kubernetes/ingress/issues/17
The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip.
a lot of this is taken care for the cloud...but NOT for bare metal.
[1] https://groups.google.com/forum/m/?utm_medium=email&utm_sour...
https://github.com/kubernetes/kubernetes/issues/27343
I was writing a blog about installing it on bare metal, but this issue got me blocked.
Though honestly i think it is a fundamental issue and not specifically related to BM.
They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean.
Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.
Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin.
We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB.
Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.
[0] https://youtu.be/4gyeixJLabo?t=814
In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
There will be a 1.5 compatible release coming soon.
Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc...
Disclosure: I work at Google on Kubernetes.
FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison.
Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.
https://github.com/kubenow/KubeNow
Tries to re-use as much as possible from great projects like Terraform, Packer, Ansible and the kubeadm tool, and just add a thin layer on top of that (less risk for bit rot), which is an approach that seems appealing to me.
https://cloud.google.com/container-engine/docs/preemptible-v...
Disclosure: I work at Google on Kubernetes.
Disclosure: I work at Google on Kubernetes.
This recent discussion had more comments on that: https://news.ycombinator.com/item?id=13085941
This works on GCE, AWS, Azure, VMWare, bare metal (via MAAS) and LXD containers. We're currently supporting 1.4 but will support 1.5 (and upgrades) in the next week or so.
Disclaimer: I'm on the team that works on this.
We have plans on including other storage vendors and mechanisms, but they aren't on the roadmap for the very near term. If end users start requesting a specific storage solution, it would go in our planning doc and get added to the roadmap. We're quite active with our early adopters that give us feedback and file bugs/requests.
To date you could continue to use alternative storage providers such as NFS or Gluster - but we don't have the PV creation+enlistment captured in the charm code just yet, but again, due to priorities. End users pretty much set the priorities for us, and we then circle back with some light weight planning and execution.
Hope this helps!
In the main kubernetes git repo, I see 39 non-merge from @canonical commits, 99% of which are under cluster/juju/* and 1 doc fix for juju, so not really a value add to Kubernetes at all. How can I, as someone deploying kubernetes on bare metal, trust you to manage a difficult project when Canonical doesn't seem to contribute much at all to the project or ecosystem?
It is great that you're making kubernetes easy to manage and deploy, but other than ease of use (which GKE does an excellent job of as well from the authors of k8s), what is Canonical's actual value add for paying you to manage k8s? Sorry for the terseness, but I'm genuinely curious what value add there is here. As a purely tech focused person, I simply don't see it. Even other firms doing exactly what you do (Apcera) have actual code change commits. I don't see a single one from Canonical that does.
GKE is great, but GKE is Google only. It's not on-prem, it's not cross cloud, and it's not portable. That's important to some people. Our contributions to cluster/juju is the distillation of our operational knowledge in running Kubernetes everywhere. The same upstream k8s, deployed with the same tooling, everywhere.
Not all value can be measured in commits :)
Thanks for your response, I was curious as to the value add and this helps.
We participate in SIGs as well, sig-on-premise being one we co-chair and co-founded. We've planning on helping the project in ways other than code contributions.
This seems to be Canonical's general direction. It just makes me sad that is is more on marketing than engineering. I wish it was more of both as clearly Canonical does Linux + marketing better than basically anyone or there wouldn't be so much Ubuntu everywhere :)
EDIT: My first response might of come off as rude, so I fixed it.
Thanks for the excellent response!
We really want these projects to have vibrant ecosystems. Canonical (and others) is contributing to that and helping users build solutions based on the existing code base. Adding features is great. Adding users is equally important.
But there's nothing stopping you from enlisting Azure PV's as a resource, Ceph managed PV's, and other incantations of durable storage. I only ask that you really consider the cost/benefit of each, and pick what makes the most sense to you.
My thoughts would be to use the azure PV disk type, and if that's not dynamic enough to meet your needs, then enlist ceph + large volumes and carve those up into RDB's to share among your workloads.
I'm sure there are others with differing opinions, and I'm happy to help you work through them (but not on a HN comment thread)
Seek me out in Slack as @lazypower, or ping me on the Juju IRC channel irc.freenode.net #juju I'm @lazypower there as well.
And finally, our juju user mailing list is another great resource for supporting questions like the above:
juju@lists.ubuntu.com
A few things to keep in mind:
These maps are service centric, and abstract units as vertical columns in their respective diagrams. Services must be HA to be considered “production ready”
Additional concerns that may/may-not be represented here:
- TLS Security on all endpoints - TLS Key Rotation in the event of compromise/upgrade/expiration - Durable storage backed workloads - ETCD state snapshots for cluster point-in-time recovery - User/RBAC - this still needs more info before i can outline it (time limited) - Network policy for namespace/application isolation (this is an unspoken requirement for many business units)
The diagrams:
Kubernetes cluster services https://docs.google.com/drawings/d/1U4GBSg9Sdn7JspoxDyA4qwGM...
Kubernetes Binary Services Topology map https://docs.google.com/drawings/d/10sXtgdelUI3GbWjrYh2z5vhF...
Kubernetes Cluster node Maps (3) https://docs.google.com/drawings/d/1x1PEE0RKvCRnP5JCAjmfbr_7...
We left off working on a Network draft diagram, and if you’re interested in contributing/participating in this process, join us in the #sig-cluster-ops slack channel. We meet thursdays (or have, new year schedule dependent)
It looks like this is going to be set to false for v1.5.1: https://github.com/kubernetes/kubernetes/pull/38708
Disclosure: I work at Google on Kubernetes.