Ask HN: How many Kubernetes clusters does your company operate?
Excluding hosting providers (e.g. AWS for EKS, Google for GKE, etc) how many Kubernetes clusters does your company operate, and what information can you share about them?
Example:
- How many k8s clusters do you have?
- How many VMs in total or typical cluster size?
- How many containers do you run?
- Why so many? (e.g. lots of customers, lots of internal developers, cell-based architecture, etc)
- Do you run on EKS, GKE, AKS, OpenShift, bare metal etc?
- How do you manage them (Rancher, Crossplane, Terraform, etc)?
- Is the platform growing or shrinking?
- What's the split between development & production?
- If you're able to share the company name, great... if not, can you explain the space you're in (e.g. SaaS provider, AdTech, etc)
- For those at scale (more than 500), what have you found that a) works well b) doesn't work well?
To get things started, here's some information I can share about my company in relation to the 2021 CNCF annual survey...
https://www.cncf.io/reports/cncf-annual-survey-2021/ (about ~2,100 responses)
About 6% of respondents had more than 5,000 nodes (we have over 50,000 - over 10x that)
About 9% of respondents ran more than 5,000 containers (we have over 500,000 - over 100x that)
About 6% of respondents ran more than 50 k8s clusters (we have over 2,000 - over 40x that)
Thanks for sharing!
2 comments
[ 5.3 ms ] story [ 13.8 ms ] threadThen each "application" gets a namespace. It can be tricky to decide which containers fit into which application exactly. It turns more into logical organization at that point.
If it's a startup / small company you can throw everything into one cluster and divide it up by namespace. Not ideal but not the end of the world.
Our other cluster houses QA/testing envs alongside our DataSci/ML workloads. That cluster can be running anywhere between 8-200 nodes, with 50-2000+ pods at any given time.
We have run into IP address allocation issues, but this mainly an issue with how we set up our VPCs.
Adtech.