BOSH is also used to deploy Cloud Foundry and a bunch of data services (RabbitMQ, Redis etc). It has the advantage of having been under continuous development for 7 years, with backends (CPIs) for most IaaSes being provided by the IaaSes themselves.
I work for Pivotal, we sponsor most of the work done on BOSH. As it happens, SAP released an experimental BOSH CPI for Kubernetes[1] and there is work underway to make that experience smoother[2].
the first time I heard BOSH in a CF related youtube video. There I thought it would mean the company Bosch and was quite surprised how advanced this company was in container technology.
That's all nice. The issue though is that BOSH has been completely leapfrogged by Kubernetes with its extensible API. Nowhere will BOSH ever get to the community reach and acceptance of the level of Kubernetes. That Boat has sailed. And with implementations of the machine specs [0,1,2,3] you get rid of the "media break" that you deem BOSH is filling (it is not). Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo?
As for the bunch of data services, I guess it's only a matter of time until you see a cambrian explosion of productive operators. I mean this non-comprehensive list [4] is already impressive.
SAP works in many projects where obligations with long term commitments have to be kept. And that is ok. BOSH CPI is one experiment to get CF on K8s. Have a look at the one which seems more attainable [6]. But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? [5]
I work for SAP in the inter-junction of SaaS, PaaS & IaaS and K8s.
It may surprise you to learn that I disagree about your core thesis. My qualifications are working for Pivotal at the inter-junction of PaaS, CaaS and FaaS. IaaS is just a hobby.
> Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo?
This is being investigated too. The main difficulty (as Brendan Burns has noted for virtual kubelet) has been that Kubernetes, ostensibly providing a smooth abstraction away from machines, actually has layer-breaking assumptions about the existence machines after all.
Cloud Foundry always had BOSH to insulate it from that concern. But BOSH-on-K8s was not super pretty in the early days, because they had overlapping concerns (mostly disks, I believe).
Kubernetes-on-BOSH is a natural fit. Standing up large distributed systems on IaaSes is BOSH's bread and butter. More to the point, that is its sole focus. Its mission is not spread amongst a cambrian explosion of alternatives (almost all of whom, you may recall, went extinct).
But in any case, it's doable. Which has the nice property that as the cluster API matures, BOSH will happily take workloads that run on VMs and run them on pods. The same experience we have today -- run an upgrade, everything is upgraded, nobody bats an eye -- will be exactly the same.
> But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend?
If you look at the community activity, the answer is pretty clearly yes: Diego can be placed behind an OPI (Project Eirini) and CF itself can run control-plane components in containers instead of VMs. Personally I am all for it.
But as you pointed out, enterprise vendors need to keep their word. Adopting Kubernetes isn't a button-press operation. We need to prove that CF-on-K8s is at least as safe and performant as CF-on-Diego has been. Your customers, and Pivotal's customers, and IBM's customers, and SUSE's customers, expect all of us to provide the roadmap and prove that it is something they can bet a company on.
As a 2y+ Bosh user and short to 2y Kubernetes user, I can only second here, that Kubernetes has outpaced Bosh. Don't get me wrong, while I would always prefer Bosh over Chef or Ansible, Kubernetes is simply the better deployment underlay.
I mean, either I have to deploy Bosh or Kubernetes (to deploy more Kubernetes clusters, because that's what we are speaking of here). Kubernetes is now commodity. The idea is to place Kubernetes control planes into Kubernetes clusters (like it became good practice to deploy the OpenStack control planes) and benefit from all the advantages Kubernetes brings along. No need to go for another tool, but there are more points:
- First off, Bosh is no thing in the Kubernetes community – it is primarily a tool used in the context of Cloud Foundry.
- By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology.
- Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes.
- VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized.
- Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA.
- Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it.
- Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back).
- Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level.
- Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS).
- Bosh packages all software into proprietary Bosh Releases and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers.
The list goes on and on and it is unlikely that Bosh can catch up again (the Pivotal IPO statement seems to indicate the same as major risk [1]). Kubernetes is the new virtualization layer and is made to run software reliably with minimal TCO. I have seen this in practice - Kubernetes is absolutely amazing - also/especially for these types of workloads.
As a 4 year BOSH user and 1 year Kubernetes user, I have to pick some nits.
> By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology.
If you're running Kubernetes on-prem, you still need to solve how to manage the Kubernetes control plane. Kubernetes-on-Kubernetes doesn't make that vanish.
> Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes.
BOSH provisions anything for which there is a CPI. Virtual machines, hardware, Kubernetes containers.
Some folks will go directly to virtual kubelets. A lot of folks won't. For on-prem they will need to solve the bootstrapping problem to get to the first install of Kubernetes. Even GKE, pretty much the gold standard public service, is still creating VMs for you.
> VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized.
And you've never over or under provisioned a container?
> Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA
BOSH is a purely operator tool. Kubernetes is not very clear about where it wants to draw the boundaries. At CF Summit a consistent theme was "we ought to use Kubernetes under the hood". At Kubecon a consistent theme was "hmmm ... maybe we should have a hood".
Scaling apps is up to whomever cares about them. On Kubernetes that's HPA or VPA or another autoscaler (I happen to work on an autoscaler that is for Kubernetes). On Cloud Foundry it's one of the several app autoscalers you can install.
> Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it.
Nobody wanted it, to be frank. Now they do and folks from Pivotal, IBM and I think SAP are working on it (as well as the cluster API).
> Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back).
Kubernetes cannot scale its components. It needs an outside tool to manage its own components. A tool like BOSH, for example.
Or, in GKE, the cluster autoscaler. Which, again, spins up VMs, not containers.
> Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level.
BOSH monitors both processes and VMs. Monit is being aggressively retired in favour of BOSH Process Manager.
> Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS)
This goes back to my point that Kubernetes has to run on something, and that it would be nice if that something is easy to manage. One way to do that is to totally relieve the operator of unnecessary toil.
The only way to ensure you have a consistent machine image is to ... have a consistent machine. Red Hat agree, which is why OpenShift has always run on RHEL. So if "lockin" means "totally opensource Linux distribution with profitable vendors who rapidly backport fixes for years in very short timeframes", you're locked in by everyone.
Our experience is that operators like the stemcell system. It's been a competitive advantage for us. Red Hat's purchase of CoreOS -- which is much more BOSH-like than Ansible -- is not a coincidence.
> Bosh packages all software into proprietary Bosh Releases
I don't mean to hijack the thread, sincere concern.
I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this kind of infrastructure. It's not clear if we are allowed to use the public cloud yet. (We are a bank, yeah, I know)
I've tried, at least:
- kubeadm
- rancher
- canonical kubernetes
- canonical kubernetes core
- some random internet recipes
And for some IaaS:
- cloudfoundry
- openstack
- cloudstack
- opennebula
- ganeti
Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc.
I've been able to hammer some of this solutions until the cluster started and had some pods or VMs running, but it feels like this are not ready for production or not for 'secure' on-premise deployment.
The deployment worked, I guess, because we had something similar to an `iso` locally to do the full install. After that, if you are not using a local registry, you get to some of the same problems I mentioned in another response.
Another problem we saw with Openshift was that it was not clear how we would be able to integrate the networking with our legacy infrastructure. If everything is working inside the cluster or, the rest of your infrastructure is kind of SOA, I guess it wouldn't be a problem, but as I mentioned... we are a bank :D
We had Red Hat consultants here for a couple of months helping our infra teams to get it stood up and functioning in our environment. It's far from trivial, especially once enterprise weirdness is figured in (I'm in healthcare, which I imaging is fairly similar to banking in this regard).
I work on the OpenStack deployment for a hedge fund where all internet access goes through a proxy. (We have the same concern about public cloud although we're slowly figuring it out.) The initial deployment predated me, but there's nothing in our config that does anything special with OpenStack and internet access, and it works. We also have a Kubernetes deployment that also (AFAIK) just works. We're not able to `docker pull` very easily because of the proxy, but if we generate our own images they work fine.
What is trying to connect to the public internet?
(If you mean that they expect a direct connection between client machines and servers, then, yes, they probably do and that's not a scenario I've run into personally.)
The Openstack deployments first failed while trying to download different pieces of sotware. As each part/script seems to use a different tool to download or get packages, we had to chase each one to find where the proxies should be configured.
Once the cluster started, again, you have to find which tool is trying to use the proxy and now can't connect with the rest of the cluster. Sorry I can't be more specific, I gave up with this some months ago. I don't think we have the scale for Openstack.
Same problems, maybe in different steps, occurred with the other tools.
To solve the `docker pull` problem, we are trying with Nexus to proxy the global registry. But given that some of the scripting/tools that deploy kubernetes do some magic starting things that we can't directly control, we can't configure docker to pull from a proxy (not that we have found how to tell the stupid thing how to avoid going to the global registry and only use our proxy/registry). Once again, when we finally managed to pull those images, we have to guess how to unconfigure proxies to be able to fully connect the cluster.
The connection between clients and servers is another concern we haven't yet got to, but we can see it won't be a simple solution.
(Oh, I don't particularly recommend OpenStack unless you know you're a fit for its scale and also the specific things it's good at, yes.)
I would expect some installer shell script not to work well, yes. If you can mirror repositories internally (or configure apt/yum/whatever to work with the proxy), that should be fine... we've deployed one OpenStack instance using (more or less) pip and one using Debian packages.
Nothing in our running infrastructure attempts to connect to the external internet.
If you get to do an AWS trial, use Kops. It works really well.
> Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc.
Same. I had to go home and do half the setup disconnected from the VPN so I could download anything without the proxy screwing it up. Something about the google storage servers on IPv4 screws with connections. I'm still waiting for the god security team to fix it.
Rancher has an "air gap" installation option, though requires you to be running your own on-premise registry (which you'll be doing if you're not sure you can use the public cloud.
I'm busy deploying 1.6 at a financial institution, and thinking about the 2.1 upgrade to Kubernetes. The Rancher team have been great so far.
I have one Firefox tab opened with that repo, haven't tried it yet in part because I'm kind of loosing hope. Starting to believe we'll have to build something ourselves.
Yes, talking to the Apprenda guys I got the impression that's their specialty: air-gapped and otherwise isolated on-premises clusters. I'm sure they'd love to get you on a support contract.
I work for Apprenda. Sure, support contracts are an option but the Kismatic Enterprise Toolkit (KET) is completely Open Source and free. And you can get community support via the Slack channel: kismatic.slack.com
I created https://github.com/cloudboss/keights because of the same issue, though it is for AWS only. I've been using it in an air gapped environment and only had to tweak two of the systemd unit dropins in the master CloudFormation template to add a proxy environment variables. The only reason I needed to do that is because two of the AWS services that are used don't yet have private VPC endpoints (cloudformation and autoscaling).
We run it in our proxy-only environment pretty regularly. You want docker to have proxy settings but not kubernetes. We use kubeadm and kubespray to install upstream. OpenShift is really broken with a proxy though and requires manual intervention to fix after the install, and even then not everything works.
It's because the kubelet is trying to downloading static pods (i.e. docker containers) from the internet. If you're using coreos, it's likely rkt is trying to download flannel (if you're using it) and the hyperkube container from the internet too.
Good news, all this is configurable. Bad news, it's no longer really a turnkey solution.
For the most part, bootstrapping kubernetes just from the hyperkube isn't that difficult. You just have to change the static manifests for the apiserver, controller, scheduler and the proxy to point to your private registry. You then need to change the kubelet(s) to use a private registry for the --pod-infra-container-image. This should probably be enough to get you up and running.
With kubeadm (1.8+) you can use imageRepository: <private-registry> in kubeadm config file.
I work in the field at Red Hat, specifically with the banks (where you are, it seems). We get OpenShift, OpenStack, and more running in prod in both air-gapped, and ssl-proxied environments with full re-encrypt. Reach out if you need help!
Like most of the others mentioned in your list, Cloud Foundry is designed to operate in disconnected environments.
I worked on buildpacks in 2014, part of my job was to ensure that it worked in disconnected environments. These days every single version of Pivotal Application Service and all of our other software products are tested together in public and private cloud configurations. For every single release[0].
We already have many large private installations at the largest banks, insurers, telcos, healthcare and automotive companies in the world. And we are able to connect you with your peers in the industry to get their views on whether to pick Cloud Foundry or an alternative.
I'm happy to put you in touch with our field folks (my email is jchester@pivotal.io) if you like.
40 comments
[ 3.4 ms ] story [ 111 ms ] threadBOSH is also used to deploy Cloud Foundry and a bunch of data services (RabbitMQ, Redis etc). It has the advantage of having been under continuous development for 7 years, with backends (CPIs) for most IaaSes being provided by the IaaSes themselves.
I work for Pivotal, we sponsor most of the work done on BOSH. As it happens, SAP released an experimental BOSH CPI for Kubernetes[1] and there is work underway to make that experience smoother[2].
[0] https://docs-cfcr.cfapps.io/
[1] https://github.com/SAP/bosh-kubernetes-cpi-release
[2] https://www.dropbox.com/s/6jv9su650a76qmq/BOSH%20Kube%20CPI-...
[1] https://xmpp.org/extensions/xep-0124.html
As for the bunch of data services, I guess it's only a matter of time until you see a cambrian explosion of productive operators. I mean this non-comprehensive list [4] is already impressive.
SAP works in many projects where obligations with long term commitments have to be kept. And that is ok. BOSH CPI is one experiment to get CF on K8s. Have a look at the one which seems more attainable [6]. But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend? [5]
I work for SAP in the inter-junction of SaaS, PaaS & IaaS and K8s.
[0] https://github.com/kubernetes-sigs/cluster-api
[1] https://github.com/kube-node/nodeset
[2] https://github.com/gardener/machine-controller-manager
[3] https://github.com/kubeup/archon
[4] https://github.com/operator-framework/awesome-operators
[5] https://www.youtube.com/watch?v=4ow7IumxkOM
[6] https://docs.google.com/document/d/1qs6UQQDWMkfOpY19XqS3CfvI...
> Maybe you could implement BOSH as implementation of the machine spec and integrate into K8s, the other way round than KuBo?
This is being investigated too. The main difficulty (as Brendan Burns has noted for virtual kubelet) has been that Kubernetes, ostensibly providing a smooth abstraction away from machines, actually has layer-breaking assumptions about the existence machines after all.
Cloud Foundry always had BOSH to insulate it from that concern. But BOSH-on-K8s was not super pretty in the early days, because they had overlapping concerns (mostly disks, I believe).
Kubernetes-on-BOSH is a natural fit. Standing up large distributed systems on IaaSes is BOSH's bread and butter. More to the point, that is its sole focus. Its mission is not spread amongst a cambrian explosion of alternatives (almost all of whom, you may recall, went extinct).
But in any case, it's doable. Which has the nice property that as the cluster API matures, BOSH will happily take workloads that run on VMs and run them on pods. The same experience we have today -- run an upgrade, everything is upgraded, nobody bats an eye -- will be exactly the same.
> But these activities are an indicator of the elephant in the room, namely CF & K8s: Will it blend?
If you look at the community activity, the answer is pretty clearly yes: Diego can be placed behind an OPI (Project Eirini) and CF itself can run control-plane components in containers instead of VMs. Personally I am all for it.
But as you pointed out, enterprise vendors need to keep their word. Adopting Kubernetes isn't a button-press operation. We need to prove that CF-on-K8s is at least as safe and performant as CF-on-Diego has been. Your customers, and Pivotal's customers, and IBM's customers, and SUSE's customers, expect all of us to provide the roadmap and prove that it is something they can bet a company on.
I mean, either I have to deploy Bosh or Kubernetes (to deploy more Kubernetes clusters, because that's what we are speaking of here). Kubernetes is now commodity. The idea is to place Kubernetes control planes into Kubernetes clusters (like it became good practice to deploy the OpenStack control planes) and benefit from all the advantages Kubernetes brings along. No need to go for another tool, but there are more points: - First off, Bosh is no thing in the Kubernetes community – it is primarily a tool used in the context of Cloud Foundry. - By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology. - Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes. - VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized. - Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA. - Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it. - Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back). - Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level. - Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS). - Bosh packages all software into proprietary Bosh Releases and doesn’t isolate the VM processes, while with Kubernetes you create individual images and run them as isolated containers.
The list goes on and on and it is unlikely that Bosh can catch up again (the Pivotal IPO statement seems to indicate the same as major risk [1]). Kubernetes is the new virtualization layer and is made to run software reliably with minimal TCO. I have seen this in practice - Kubernetes is absolutely amazing - also/especially for these types of workloads.
[1] https://techcrunch.com/2018/03/23/pivotal-software-files-for...
> By using Kubernetes to manage itself you have less components, less learning. It is there already and everybody who works with Kubernetes doesn’t need to pick up yet another technology.
If you're running Kubernetes on-prem, you still need to solve how to manage the Kubernetes control plane. Kubernetes-on-Kubernetes doesn't make that vanish.
> Bosh provisions VMs, which are slow to provision. What takes minutes with Bosh, takes seconds with Kubernetes.
BOSH provisions anything for which there is a CPI. Virtual machines, hardware, Kubernetes containers.
Some folks will go directly to virtual kubelets. A lot of folks won't. For on-prem they will need to solve the bootstrapping problem to get to the first install of Kubernetes. Even GKE, pretty much the gold standard public service, is still creating VMs for you.
> VMs lead to massive fragmentation as they are mostly under-utilized and in the worst case over-utilized.
And you've never over or under provisioned a container?
> Bosh can’t scale processes as it commandeers only VMs, while in Kubernetes everything is a pod/container that can scale up or out automatically using the VPA/HPA
BOSH is a purely operator tool. Kubernetes is not very clear about where it wants to draw the boundaries. At CF Summit a consistent theme was "we ought to use Kubernetes under the hood". At Kubecon a consistent theme was "hmmm ... maybe we should have a hood".
Scaling apps is up to whomever cares about them. On Kubernetes that's HPA or VPA or another autoscaler (I happen to work on an autoscaler that is for Kubernetes). On Cloud Foundry it's one of the several app autoscalers you can install.
> Using the Bosh Kubernetes CPI has other disadvantages we can discuss in breadth, but I don't think we should, because its experimental at best and I know no installation actually running on it.
Nobody wanted it, to be frank. Now they do and folks from Pivotal, IBM and I think SAP are working on it (as well as the cluster API).
> Bosh can’s scale VMs automatically, while Kubernetes can. If the cluster is nearing saturation, the cluster-autoscaler provisions more VMs on-the-fly (or shrinks the cluster back).
Kubernetes cannot scale its components. It needs an outside tool to manage its own components. A tool like BOSH, for example.
Or, in GKE, the cluster autoscaler. Which, again, spins up VMs, not containers.
> Bosh uses outdated Monit to watch over single processes, while Kubernetes pods are managed by the container engine on the lower level and by Kubernetes itself on the higher level.
BOSH monitors both processes and VMs. Monit is being aggressively retired in favour of BOSH Process Manager.
> Bosh depends on rather large Ubuntu-based or CentOS Pivotal stemcells (with the Bosh agent) and Pivotal updating them, while there are much light-weighter options like CoreOS Container Linux and no lock-in (you can take almost any OS)
This goes back to my point that Kubernetes has to run on something, and that it would be nice if that something is easy to manage. One way to do that is to totally relieve the operator of unnecessary toil.
The only way to ensure you have a consistent machine image is to ... have a consistent machine. Red Hat agree, which is why OpenShift has always run on RHEL. So if "lockin" means "totally opensource Linux distribution with profitable vendors who rapidly backport fixes for years in very short timeframes", you're locked in by everyone.
Our experience is that operators like the stemcell system. It's been a competitive advantage for us. Red Hat's purchase of CoreOS -- which is much more BOSH-like than Ansible -- is not a coincidence.
> Bosh packages all software into proprietary Bosh Releases
This is flatly wrong.
Releases ar...
I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this kind of infrastructure. It's not clear if we are allowed to use the public cloud yet. (We are a bank, yeah, I know)
I've tried, at least:
- kubeadm
- rancher
- canonical kubernetes
- canonical kubernetes core
- some random internet recipes
And for some IaaS: - cloudfoundry
- openstack
- cloudstack
- opennebula
- ganeti
Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc.
I've been able to hammer some of this solutions until the cluster started and had some pods or VMs running, but it feels like this are not ready for production or not for 'secure' on-premise deployment.
The deployment worked, I guess, because we had something similar to an `iso` locally to do the full install. After that, if you are not using a local registry, you get to some of the same problems I mentioned in another response.
Another problem we saw with Openshift was that it was not clear how we would be able to integrate the networking with our legacy infrastructure. If everything is working inside the cluster or, the rest of your infrastructure is kind of SOA, I guess it wouldn't be a problem, but as I mentioned... we are a bank :D
What is trying to connect to the public internet?
(If you mean that they expect a direct connection between client machines and servers, then, yes, they probably do and that's not a scenario I've run into personally.)
Once the cluster started, again, you have to find which tool is trying to use the proxy and now can't connect with the rest of the cluster. Sorry I can't be more specific, I gave up with this some months ago. I don't think we have the scale for Openstack.
Same problems, maybe in different steps, occurred with the other tools.
To solve the `docker pull` problem, we are trying with Nexus to proxy the global registry. But given that some of the scripting/tools that deploy kubernetes do some magic starting things that we can't directly control, we can't configure docker to pull from a proxy (not that we have found how to tell the stupid thing how to avoid going to the global registry and only use our proxy/registry). Once again, when we finally managed to pull those images, we have to guess how to unconfigure proxies to be able to fully connect the cluster.
The connection between clients and servers is another concern we haven't yet got to, but we can see it won't be a simple solution.
I would expect some installer shell script not to work well, yes. If you can mirror repositories internally (or configure apt/yum/whatever to work with the proxy), that should be fine... we've deployed one OpenStack instance using (more or less) pip and one using Debian packages.
Nothing in our running infrastructure attempts to connect to the external internet.
> Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc.
Same. I had to go home and do half the setup disconnected from the VPN so I could download anything without the proxy screwing it up. Something about the google storage servers on IPv4 screws with connections. I'm still waiting for the god security team to fix it.
I'm busy deploying 1.6 at a financial institution, and thinking about the 2.1 upgrade to Kubernetes. The Rancher team have been great so far.
https://rancher.com/docs/rancher/v2.x/en/installation/air-ga...
It just works...
It's because the kubelet is trying to downloading static pods (i.e. docker containers) from the internet. If you're using coreos, it's likely rkt is trying to download flannel (if you're using it) and the hyperkube container from the internet too.
Good news, all this is configurable. Bad news, it's no longer really a turnkey solution.
For the most part, bootstrapping kubernetes just from the hyperkube isn't that difficult. You just have to change the static manifests for the apiserver, controller, scheduler and the proxy to point to your private registry. You then need to change the kubelet(s) to use a private registry for the --pod-infra-container-image. This should probably be enough to get you up and running.
With kubeadm (1.8+) you can use imageRepository: <private-registry> in kubeadm config file.
http://v1.uncontained.io/playbooks/installation/#design-for-...
This could be used to deploy OpenShift Origin, the upstream of the enterprise version
I worked on buildpacks in 2014, part of my job was to ensure that it worked in disconnected environments. These days every single version of Pivotal Application Service and all of our other software products are tested together in public and private cloud configurations. For every single release[0].
We already have many large private installations at the largest banks, insurers, telcos, healthcare and automotive companies in the world. And we are able to connect you with your peers in the industry to get their views on whether to pick Cloud Foundry or an alternative.
I'm happy to put you in touch with our field folks (my email is jchester@pivotal.io) if you like.
[0] https://content.pivotal.io/blog/you-deserve-a-continuously-i...
Start the conversation now with your networking folks about a more transparent setup.
It was for a proof of concept project, so I don’t know how it would perform in production but it was promising.