> To be fair, Kubernetes right now only schedules relatively small clusters. This is not really true: https://news.ycombinator.com/item?id=25907312 https://cloud.google.com/blog/products/containers-kubernetes...…
> it throws out all of the interesting parts of Borg This is not true. It throws out the Google-specific parts of Borg (like integration with Google's service discovery, load balancing, and monitoring systems) and…
> The biggest is going to be utilisation. Combining workloads pools variance, meaning you can safely run at a higher baseline load. Utilization is arguably the biggest benefit (fewer nodes if you can share nodes among…
I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota,…
A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this…
The rescheduler in Borg isn't a scheduler -- it just evicts pods, and then they go into the regular scheduler's pending queue and the regular scheduler decides where to schedule them. (At least that's how it worked at…
> Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so…
> It can only span multiple clouds now because other clouds had to ship Kubernetes. This isn't true. People were running open-source Kubernetes on AWS and Azure before either provider had a hosted Kubernetes service. In…
You can actually build an Omega vertical / Mesos framework architecture on Kubernetes, as described in this doc[1]. That doc pre-dated CRDs; the way you'd do it today is to build the application lifecycle management…
Stubby and Chubby are not related to Borg's scalability. The reason Kubernetes scalability was originally not so great was because it simply wasn't prioritized. We were more concerned with building a feature set that…
[Disclaimers: I worked on Borg and Omega, and currently work on Kubernetes/GKE. Everything here is my personal opinion.] There's a lot to unpack here, but I'll do my best. I don't see Kubernetes locking people into GKE.…
I don't understand why you say "Having containers with local volumes is counter productive." I would agree it's probably not a good architecture if you're running a huge single-node Oracle database, but it's an…
> I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a…
Kubernetes supports pluggable schedulers: http://kubernetes.io/docs/admin/multiple-schedulers/ It also supports pluggable "controllers," which manage the lifecycle of containers. (So a Kubernetes scheduler + controller…
Regarding packages, you might be interested in https://github.com/kubernetes/charts Disclosure: I work on the Kubernetes project at Google.
We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here:…
> To be fair, Kubernetes right now only schedules relatively small clusters. This is not really true: https://news.ycombinator.com/item?id=25907312 https://cloud.google.com/blog/products/containers-kubernetes...…
> it throws out all of the interesting parts of Borg This is not true. It throws out the Google-specific parts of Borg (like integration with Google's service discovery, load balancing, and monitoring systems) and…
> The biggest is going to be utilisation. Combining workloads pools variance, meaning you can safely run at a higher baseline load. Utilization is arguably the biggest benefit (fewer nodes if you can share nodes among…
I don't think multi-tenancy has been "retrofitted" onto Kubernetes. Kubernetes was designed with multi-tenancy in mind from the very early releases -- namespaces, authn/authz (initially ABAC, later RBAC), ResourceQuota,…
A pattern we're seeing a lot of recently is one cluster per "stage" per region, where a "stage" is something like dev/test, canary, and prod. (In some cases only prod is replicated across multiple regions.) I think this…
The rescheduler in Borg isn't a scheduler -- it just evicts pods, and then they go into the regular scheduler's pending queue and the regular scheduler decides where to schedule them. (At least that's how it worked at…
> Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so…
> It can only span multiple clouds now because other clouds had to ship Kubernetes. This isn't true. People were running open-source Kubernetes on AWS and Azure before either provider had a hosted Kubernetes service. In…
You can actually build an Omega vertical / Mesos framework architecture on Kubernetes, as described in this doc[1]. That doc pre-dated CRDs; the way you'd do it today is to build the application lifecycle management…
Stubby and Chubby are not related to Borg's scalability. The reason Kubernetes scalability was originally not so great was because it simply wasn't prioritized. We were more concerned with building a feature set that…
[Disclaimers: I worked on Borg and Omega, and currently work on Kubernetes/GKE. Everything here is my personal opinion.] There's a lot to unpack here, but I'll do my best. I don't see Kubernetes locking people into GKE.…
I don't understand why you say "Having containers with local volumes is counter productive." I would agree it's probably not a good architecture if you're running a huge single-node Oracle database, but it's an…
> I don't want to break your hopes but stateful containers will only ever run on GCE and AWS. Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a…
Kubernetes supports pluggable schedulers: http://kubernetes.io/docs/admin/multiple-schedulers/ It also supports pluggable "controllers," which manage the lifecycle of containers. (So a Kubernetes scheduler + controller…
Regarding packages, you might be interested in https://github.com/kubernetes/charts Disclosure: I work on the Kubernetes project at Google.
We're working on directly-integrated Spark-on-Kubernetes right now and would love to get input from folks who are interested. The Github issue where we're discussing it is here:…