I've been thinking about what we need to simplify distributed applications for the past few years, and we really need reusable components[1]. Anything we can do to make containers more consistent and easier to build is important.
The ipsec implementation just configures ipsec in the kernel using a go netlink library[1]. It is similar to how it would be accomplished using iproute2 via `ip xfrm policy` and `ip xfrm state`.
I applaud the exploration, though the author appears to class everything outside of the container as a 'communications layer' and vaguely ascribe knowledge of the credentials and processes necessary to facilitate orchestration, configuration management, etc. across multiple containers to this area. This is not solving the problem, only hand-waving it elsewhere.
The potential use of a container-initiated networking event (socket open) to trigger responses at the infrastructure-level is the key architectural novelty here. This is essentially a suggestion for 'implicit infrastructure' as opposed to 'explicit infrastructure'; with all of the benefits and drawbacks you would expect from such a paradigm shift.
Personally I believe that the implicit paradigm is mostly useful in specific, relatively controlled use cases, such as service testing or behavioral profiling prior to live deployment.
Explicit infrastructure is that which is completely specified and instantiated and ahead of use. This would include resource allocation and security policy for computational, storage and networking infrastructure. Building infrastructure in the pre-cloud/pre-virt era generally necessitated this approach.
Implicit infrastructure is constructed reactively, over time, in response to evolving stimuli but also based upon dependencies or requirements specified or assumed. For instance, today this occurs in automated scale-out on some cloud systems. The author describes a conceptually similar (from the infrastructure specification and management standpoint) mode of operation here by suggesting the creation of new nodes in response to socket opens (networking events) on individual containers.
Ultimately, in the latter case you are giving up some degree of control and known state in favour of flexibility. This makes sense for some scenarios, but greatly complicates things in others. Both are valid but the viable domain of the latter is more limited.
I understand, that sounds very close to what I've been working on, a functionally reactive infrastructure that is configured with constraint based logic DSL that starts from QoS constraints. The QoS constraints allows one to specify goals, and allow infrastructure to be built up implicitly.
I've read some of the stuff on your website, it's very closely aligned to what I'm working on. We should talk in more detail.
FWIW I agree that reactive infrastructure is fairly limited in scope. The traditional model of infrastructure is everything running in a single server. If your unit of deployment is a vm, then orchestration, service discovery, security policy must be done within the vm image or via a configuration management system. If your unit of deployment is a container, then these pieces can be done outside of the container system. A standard interface for the way the container communicates with the outside world allows these systems to be shared and reused and perhaps one day standardized.
I agree there's a move towards reuse. I disagree there's any fundamental difference between hypervisor-hosted VMs and containers as far as infrastructure, connectivity, configuration management, monitoring and all the rest goes. I also disagree this is a meaningful step forward for a general use case due to paradigm quirks, but I applaud the exploration.
Vms and containers are roughly interchangable, yes, but I would argue that the tendency for vms to be treated as whole servers leads deployers to pack all of the solutions to the above problems into the vm building block. In the end there is a tightly-coupled virtual appliance and it is difficult to reuse the useful pieces. We missed the opportunity to separate these concerns and have meaningful and reusable pieces.
Containers are giving us another opportunity to revisit these choices. I'm hoping we don't miss the opportunity again.
I think this is a key point to bring to light, because most of the hype around using containers seems to focus on the fact that you can replace the hypervisor layer with a container manager (docker) and achieve a ton of benefit. For most use cases there isn't a significant enough difference to matter.
Changing the paradigm for how applications are built and deployed is the key innovation that docker is offering us, but as you point out, there are still plenty of problems to solve in this area.
I agree that people are lazier at embracing change when building with VMs... then again, they might be a different audience, too. I also agree with the rest of your post.
The point of hand waving orchestration, configuration management, etc. is to separate the concern of what is running from the concern of how it is connected to other things. My hope is that container builders could simply focus on dependencies and application code and that the act of connecting components together can be handled by a different layer in the system.
Sure, it's just that there's no solution on the other side of the hand-waving ... and your portal to the other side is also a non-standard one with limitations. As I said, I think the exploration is meaningful and the paradigm has merit but it's hard to target general applicability with such an approach.
I totally agree. This is just one of the options I outline for standardizing networking in my blog post[1] and this one probably involves a little too much magic to be generally applicable.
It is interesting to note that most of the frameworks that are trying to solve the the rest of the problem have moved along similar lines. For example Kubernetes provides both the ability to launch containers in the same namespace and also provides a proxy. Openshift uses iptables rewriting via GearD.
Your comments are very reasonable, but I would caution against reading much in to the current state of open source offerings. I've tried different approaches to resolving the networking side myself and concluded that there is no one true solution here and therefore it must be left out of service scope (ie. abstracted adequately).
This is particularly the case if portability is required, because different business-level requirements call for different service and infrastructure topologies utilizing different technologies, not all of which actually support any single approach. Think hypervisor-based VMs, containers, bare-metal clusters, etc.
Therefore I believe that solutions like docker and kubernetes are - given present architecture as I understand them - critically misaligned, in that they do not go far enough to abstract some target infrastructure related questions away from services. To do so would require reshuffling their APIs significantly and is thus unlikely to spontaneously occur in the near-term without concerted effort.
More recent iterations of the approach I have been taking assume that hypervisors, containers, jails, JVM guests or embedded systems firmware (eg. Android) should all be conceptually in-scope at the level of service authors. When considering the target deployment environment, networking is optionally in scope, and then only optionally TCP/IP. Given this redefined perspective, I believe that if a minimalist, process-oriented solution developed to facilitate the SaaS/unix SOA market is not robust enough to serve at least this range of alternative development trajectories, then in my view it is probably making invalid assumptions somewhere along the line. Of course, providing so general a tool usefully and without irritating developers or throwing up barriers to adoption is a real challenge.
This is very insightful. It is very hard to produce something that provides clear value in the microcosm (i.e. useful to developers) but is also valuable in the macrocosm (i.e. can be deployed to all targets).
One approach is to build something that solves the small problem but keeps enough options open in order to eventually grow to encompass the larger issues. I think this generally superior to the reverse approach, which tends to die from lack of adoption.
I don't know if docker, kubernetes, et. al. will actually get there. I agree that they will have to reshuffle and rethink to make it happen, but I hope that they can.
18 comments
[ 4.6 ms ] story [ 42.2 ms ] threadI've been thinking about what we need to simplify distributed applications for the past few years, and we really need reusable components[1]. Anything we can do to make containers more consistent and easier to build is important.
[1] https://medium.com/@vishvananda/standard-components-not-stan...
[1] https://github.com/vishvananda/netlink
The potential use of a container-initiated networking event (socket open) to trigger responses at the infrastructure-level is the key architectural novelty here. This is essentially a suggestion for 'implicit infrastructure' as opposed to 'explicit infrastructure'; with all of the benefits and drawbacks you would expect from such a paradigm shift.
Personally I believe that the implicit paradigm is mostly useful in specific, relatively controlled use cases, such as service testing or behavioral profiling prior to live deployment.
More thoughts on the same: http://stani.sh/walter/pfcts
Implicit infrastructure is constructed reactively, over time, in response to evolving stimuli but also based upon dependencies or requirements specified or assumed. For instance, today this occurs in automated scale-out on some cloud systems. The author describes a conceptually similar (from the infrastructure specification and management standpoint) mode of operation here by suggesting the creation of new nodes in response to socket opens (networking events) on individual containers.
Ultimately, in the latter case you are giving up some degree of control and known state in favour of flexibility. This makes sense for some scenarios, but greatly complicates things in others. Both are valid but the viable domain of the latter is more limited.
I've read some of the stuff on your website, it's very closely aligned to what I'm working on. We should talk in more detail.
Containers are giving us another opportunity to revisit these choices. I'm hoping we don't miss the opportunity again.
I think this is a key point to bring to light, because most of the hype around using containers seems to focus on the fact that you can replace the hypervisor layer with a container manager (docker) and achieve a ton of benefit. For most use cases there isn't a significant enough difference to matter.
Changing the paradigm for how applications are built and deployed is the key innovation that docker is offering us, but as you point out, there are still plenty of problems to solve in this area.
It is interesting to note that most of the frameworks that are trying to solve the the rest of the problem have moved along similar lines. For example Kubernetes provides both the ability to launch containers in the same namespace and also provides a proxy. Openshift uses iptables rewriting via GearD.
[1] https://medium.com/@vishvananda/standard-components-not-stan...
This is particularly the case if portability is required, because different business-level requirements call for different service and infrastructure topologies utilizing different technologies, not all of which actually support any single approach. Think hypervisor-based VMs, containers, bare-metal clusters, etc.
Therefore I believe that solutions like docker and kubernetes are - given present architecture as I understand them - critically misaligned, in that they do not go far enough to abstract some target infrastructure related questions away from services. To do so would require reshuffling their APIs significantly and is thus unlikely to spontaneously occur in the near-term without concerted effort.
More recent iterations of the approach I have been taking assume that hypervisors, containers, jails, JVM guests or embedded systems firmware (eg. Android) should all be conceptually in-scope at the level of service authors. When considering the target deployment environment, networking is optionally in scope, and then only optionally TCP/IP. Given this redefined perspective, I believe that if a minimalist, process-oriented solution developed to facilitate the SaaS/unix SOA market is not robust enough to serve at least this range of alternative development trajectories, then in my view it is probably making invalid assumptions somewhere along the line. Of course, providing so general a tool usefully and without irritating developers or throwing up barriers to adoption is a real challenge.
One approach is to build something that solves the small problem but keeps enough options open in order to eventually grow to encompass the larger issues. I think this generally superior to the reverse approach, which tends to die from lack of adoption.
I don't know if docker, kubernetes, et. al. will actually get there. I agree that they will have to reshuffle and rethink to make it happen, but I hope that they can.