There are several differences:
1. We're coupled with the kernel and operating at layer 4. Specifically, we play with the conntrack subsystem. This means, you may pay a little higher cost for the first packet of a connection (because we've got to program the flow), but once the flow is programmed, it's totally going through kernel space. The way Minuteman is meant to be used is to act as the underlying router of a connection pool, or some such.
2. We expose the pools as "VIPs" or Virtual IPs. When you expose a service, you specify an IP-port pair (like 1.2.3.4:5000). In my opinion, that's a little bit nicer than Bamboo's routing.
1. This is largely a restriction that comes from earlier versions. In earlier versions we relied on the source port to be in the range, as we remapped it in user space. Now, we let iptables / conntrack do it. We might go back to writing it with the conntrack entry though. There are some unfortunate awkward aspects of how iptables does free tulle mapping that can prove annoying.
2. This is a complicated question. You don't pay minuteman for active connections, only at connection construction time. You are reliant on netfilter's conntrack capabilities. By default these are built for 100s of thousands of connections. Given that in DC/OS this runs on every client, we are fairly confident in this approach, though conntrack could reasonably scale to millions of connections.
I really enjoy naming things. The naming has a bit of a history. The first version used open flow. We wrote the flows on demand for the connections, and the project's name was 'Minuteman: flows at a moment's notice' as a reference to the soldiers during the revolutionary war that were prepared to fight at a moment's notice.
Although we changed the architecture, we decided to stick with Minuteman. Mesosphere's naming scheme has historically had to do with space. The road to the stars was originally paved by programs such as Minuteman -- Our mission is to orchestrate the data center. You can't do that without networking. Minuteman is our first networking component of many as one of the first components to help pave our way.
"You must run a stock kernel from RHEL 7.2+, or Ubuntu 14.04+ LTS"
Are there any particular features or patches that are required? I maintain the Mesos package in NixOS, along with our patches for Ubuntu Fan Networking, and I would be interested in playing around with this.
This should probably more prominently explain at the beginning of the README that it's for Linux. Nothing wrong with that, but being bound to Linux is an important detail.
Thanks for the response. Agree about the Bamboo routing. Lashup is interesting. Is the conntrack data replicated then to all Minuteman nodes so if you were to lose one minutemen a client wouldn't lose its "established" state when it has to connect to another minuteman node? Or am I failing to understand the design?
20 comments
[ 0.21 ms ] story [ 49.0 ms ] threadhttps://github.com/QubitProducts/bamboo
So it seem the kernel is doing most of the work to handle the connection. The load balancer isn't really proxying the connections.
There are several differences: 1. We're coupled with the kernel and operating at layer 4. Specifically, we play with the conntrack subsystem. This means, you may pay a little higher cost for the first packet of a connection (because we've got to program the flow), but once the flow is programmed, it's totally going through kernel space. The way Minuteman is meant to be used is to act as the underlying router of a connection pool, or some such.
2. We expose the pools as "VIPs" or Virtual IPs. When you expose a service, you specify an IP-port pair (like 1.2.3.4:5000). In my opinion, that's a little bit nicer than Bamboo's routing.
3. We work with all Mesos frameworks.
4. We have an underlying distributed control plane (https://github.com/dcos/lashup), which lets us detect failures incredibly fast. We've evaluated its capabilities here: https://github.com/dcos/minuteman#evaluation
2. What is the maximum limit on active connection?
2. This is a complicated question. You don't pay minuteman for active connections, only at connection construction time. You are reliant on netfilter's conntrack capabilities. By default these are built for 100s of thousands of connections. Given that in DC/OS this runs on every client, we are fairly confident in this approach, though conntrack could reasonably scale to millions of connections.
Although we changed the architecture, we decided to stick with Minuteman. Mesosphere's naming scheme has historically had to do with space. The road to the stars was originally paved by programs such as Minuteman -- Our mission is to orchestrate the data center. You can't do that without networking. Minuteman is our first networking component of many as one of the first components to help pave our way.
Are there any particular features or patches that are required? I maintain the Mesos package in NixOS, along with our patches for Ubuntu Fan Networking, and I would be interested in playing around with this.
We've thought about what it would take to port it to Windows, and it's feasible. Just a matter of work. If you're interested, let me know.
Like the use of a in-kernel netfilter for data plane.
Clustering and failure recovery look solid as well.