Fantastic examples. I appreciate that they allude to the (canonical?) example of how Nginx does zero downtime upgrades using SO_REUSEPORT. I had hard of the trick of sharing the FD with a forked process but it's nice to a working example.
This is an excellent write up, very easy to follow. Overseer [1] is a pretty good implementation as well.
"Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process."
Even in that case you don't need graceful in-place restarts. If you have multiple replicas of the load balancer and ECMP from higher on your stack, all you need is for your balancer process to 1) remove itself as a destination for new flows; 2) stop accepting new connections and complete all existing connections; 3) shut down and restart; 4) add itself back to the set of new flow destinations.
If you don't have multiple instances of your load balancer, that's weird.
I don't believe this approach would work for a layer 7 load balancer (where the connection is terminated on the host). This is in regards to the original article, which is discussing how to maintain socket state through a process upgrade.
When you remove the host from the routing table, the ECMP algorithm will redistribute the traffic to the other hosts still in the pool. For TCP connections, this will generate reset's, and drop any active connections moved between hosts. Also, at times I've not gotten clear answers from router vendors on whether they support a consistent hash in their ECMP implementation, which means that connections through the load balancers not being reloaded may also be rebalanced to other hosts and be affected by the routing table change.
Even if we consider going down the stack to a layer 4 load balancer, outside the scope of this article, it may still be desirable to combine approaches and offer hitless reloads (a common case) and not just recover from a node failure (a less common case).
The reason is, that you would need state replication of the load balancing tables to support removing the load balancer from the routing tables, and having another load balancer take over it's work while maintaining the connections through the load balancer. This is under the assumption that the service being load balanced requires an uninterrupted connection of course.
Disclaimer: I work with the author of the article and contributed to it.
I've worked with ECMP routers where you can inform that router you'd like new flows to be sent elsewhere while existing flows continue to be sent to their established endpoints. JunOS for example has this feature.
This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.
> Once you have that problem you can forget about graceful in-place restarts forever.
Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.
There are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine
Every service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.
I would say this approach is for situation where the model you're using isn't always applicable.
In the case of this article, it's discussing teleport, a drop in replacement SSH server. For many users, this is their administrative connection to a fleet of servers (teleport can operate as a transparent bastion, as well as on the end host). So in our case, it's a desirable feature to not interrupt existing shell sessions, while replacing the software serving those connections. This could mean the admin's own connection to the server doesn't fail mid-upgrade, or that hundreds of internal employee's don't groan because whatever they were doing that used a shell just got dropped and they need to restart.
Disclaimer: I work with the author of the original article, and contributed to the article.
I don't take any issue with treating this as a special case scenario, although I think the modern trend is to treat the need for long-lived SSH sessions for managing services as a design smell.
What I do take issue with is the suggestion that handling such special cases is "table stakes." A little humility goes a long way in bolstering credibility.
I agree. The characterization of live reload capability as "table stakes" for availability is nonsense. High-availability designs never require singleton services.
Being able to replace vulnerable singletons that require process and TCP socket gymnastics with multiple instances having good fault isolation properties is the real "table stakes" for modern highly-available service architectures.
The fact that I disagree with your characterization or opinions doesn't mean I didn't read or comprehend the article. Stateful protocols on the Internet are few as a proportion of overall traffic, and it was a wise choice for HTTP to be stateless. This improves its ability to be highly available.
Teleport and SSH themselves are absolutely singletons in the most common use case: you want to SSH to the designated server, and that server alone. The service is "ssh on instance X." No other server is an acceptable substitute. That fact makes the service a singleton.
Listen, don't get me wrong -- I'm very pleased that you put this article out as it's very insightful. The technical information in it speaks for itself; it didn't need to be burdened with controversial opinion.
Perhaps the article was not explaining in detail what this daemon does. No, it's not just an SSH endpoint on instance X. For large deployments it is an SSH gateway (better implementation of jumphost) to your entire data center and it's running on multiple hosts behind an LB. Yet, every instance of it may be supporting hundreds of long-lived SSH connections originating from multiple teams, some of them can be deploying something critically important.
So, yes, this socket acrobatics is absolutely table stakes and is required (and has been requested by corporate customers of T) for robust infrastructure upgrades. Old-timers like to refer to this as "carrier-grade" way of doing things.
You may dislike SSH itself, but hey - that's what our current UNIX tooling depends on, so we have to make it work, and work well. Besides, even stateless HTTP sometimes needs reliable long-lived connections! Full disclaimer: I'm one of early Teleport contributors.
I don't dislike ssh. It's sometimes necessary and useful. But the modern way of performing deployments is not via ssh -- it's via container and load balancer orchestration and virtual machine management infrastructure.
Making your customers happy is good for business, and I salute you for the work you've done in this area. But I still take issue with your "table stakes" characterization and maintain that it is best omitted from an otherwise good discussion. Being "table stakes" for your business is different from being "table stakes" for service architectures generally.
You know that some people work on servers that are stateful? Not everyone use REST type of transport where a failed node is ok because the lb will route traffic to another place. Your solution doesn't work for things like game servers.
Even stateful services like game servers could be organised to handle losing a node. It's harder to do and you need some state replication in place. (versioned state if you want to do upgrades that way) It may not be cost effective to do so. But the possibility is there.
I worked on a plan to have live migration of multiparty phone calls between data centres in realtime. The tech is available. But your planning outcome may be: it's cheaper to make single node more stable and deal with the occasional failure.
"Stateful" only means you need to care about the state being moved. Things like Java hotswap, Erlang live code reload, and your custom solutions can handle that.
My usecase is that I have 1500+ raspberry pi class machines running my daemon process (golang agent). I want to be able to upgrade the binary, without downtime.
Thanks for the suggestion. It seems like Nerves needs a whole firmware to support it. These are machines where the most I can do is copy a binary onto them (crypto mining hardware). golang was the simplest solution for cross compiling I could find at the time.
It's theoretically interesting to hand off a listen socket, but I'd rather take 1/n machines out of service than leave half of my memory unused on every machine.
Edit: I also remember hearing some out-there stuff about handing off your IP address and open connection state to another machine so the client doesn't even notice…
I like to use a simple protocol: 1) Server receives shutdown signal 2) Server tells all clients to disconnect, and also tells the load balancer to not send more traffic 3) Client's connect to another server or waits for the server to restart 4) server exit
How do you tell the load balancer to stop sending traffic? I've done two ways in previous jobs:
- Refuse all new connections. If you can change the program, this can be done by closing the listening socket (I think) , but I just installed an iptables rule.
- use the load balancer's API to offline the server(at the time it was F5).
I liked the first solution because it was load balancer agnostic, but required retries by the load balancer or client to avoid disruptions. Curious to hear what other people did.
See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending that job traffic. The concept of lameduck is you report yourself as unhealthy but are still available to serve traffic, you wait before all of your clients that were loaded balanced to you before you stop a sending traffic.
I think it's best if the client can handle the load balancing by connecting to a random server, and have some servers built into the source for bootstrapping. The client can know what to do if the connection closes and handle interruptions.
I'm using socketmaster[1] to do the graceful update or just normal service reload. It is basically another process to facilitate socket exchange between 2 process. Just make sure that the service able to shutdown gracefully.
However this approach need the binary to be started from socketmaster (as child process). I'm not sure what is the real cons tho, except the fact that it alter SIGHUP behavior - which is fine in this case imo.
Lol, no. Modern, professional software development uses a load balancer to manage a canary/red-black deployment strategy. Zero downtime deployments to a live server, outside of telecom/finance, is a sign of an amateurish team.
46 comments
[ 3.6 ms ] story [ 86.1 ms ] thread"Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process."
[1] https://github.com/jpillora/overseer
https://github.com/crawshaw/littleboss
If you don't have multiple instances of your load balancer, that's weird.
When you remove the host from the routing table, the ECMP algorithm will redistribute the traffic to the other hosts still in the pool. For TCP connections, this will generate reset's, and drop any active connections moved between hosts. Also, at times I've not gotten clear answers from router vendors on whether they support a consistent hash in their ECMP implementation, which means that connections through the load balancers not being reloaded may also be rebalanced to other hosts and be affected by the routing table change.
Even if we consider going down the stack to a layer 4 load balancer, outside the scope of this article, it may still be desirable to combine approaches and offer hitless reloads (a common case) and not just recover from a node failure (a less common case).
The reason is, that you would need state replication of the load balancing tables to support removing the load balancer from the routing tables, and having another load balancer take over it's work while maintaining the connections through the load balancer. This is under the assumption that the service being load balanced requires an uninterrupted connection of course.
Disclaimer: I work with the author of the article and contributed to it.
"Truly Seamless Reloads with HAProxy – No More Hacks!"
https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...
1: https://golang.org/pkg/os/exec/#Cmd
Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.
In the case of this article, it's discussing teleport, a drop in replacement SSH server. For many users, this is their administrative connection to a fleet of servers (teleport can operate as a transparent bastion, as well as on the end host). So in our case, it's a desirable feature to not interrupt existing shell sessions, while replacing the software serving those connections. This could mean the admin's own connection to the server doesn't fail mid-upgrade, or that hundreds of internal employee's don't groan because whatever they were doing that used a shell just got dropped and they need to restart.
Disclaimer: I work with the author of the original article, and contributed to the article.
What I do take issue with is the suggestion that handling such special cases is "table stakes." A little humility goes a long way in bolstering credibility.
Being able to replace vulnerable singletons that require process and TCP socket gymnastics with multiple instances having good fault isolation properties is the real "table stakes" for modern highly-available service architectures.
...
> replace vulnerable singletons
You clearly didn't read (or failed to comprehend) the article. Not every protocol on the internet is stateless and no, Teleport is not a singleton.
Teleport and SSH themselves are absolutely singletons in the most common use case: you want to SSH to the designated server, and that server alone. The service is "ssh on instance X." No other server is an acceptable substitute. That fact makes the service a singleton.
Listen, don't get me wrong -- I'm very pleased that you put this article out as it's very insightful. The technical information in it speaks for itself; it didn't need to be burdened with controversial opinion.
So, yes, this socket acrobatics is absolutely table stakes and is required (and has been requested by corporate customers of T) for robust infrastructure upgrades. Old-timers like to refer to this as "carrier-grade" way of doing things.
You may dislike SSH itself, but hey - that's what our current UNIX tooling depends on, so we have to make it work, and work well. Besides, even stateless HTTP sometimes needs reliable long-lived connections! Full disclaimer: I'm one of early Teleport contributors.
Making your customers happy is good for business, and I salute you for the work you've done in this area. But I still take issue with your "table stakes" characterization and maintain that it is best omitted from an otherwise good discussion. Being "table stakes" for your business is different from being "table stakes" for service architectures generally.
I worked on a plan to have live migration of multiparty phone calls between data centres in realtime. The tech is available. But your planning outcome may be: it's cheaper to make single node more stable and deal with the occasional failure.
"Stateful" only means you need to care about the state being moved. Things like Java hotswap, Erlang live code reload, and your custom solutions can handle that.
See http://erlang.org/doc/design_principles/release_handling.htm... for the basics. With no shared state, and no global variables, and no side effects, you can update functions so the next time they're called they're something else.
Edit: I also remember hearing some out-there stuff about handing off your IP address and open connection state to another machine so the client doesn't even notice…
- Refuse all new connections. If you can change the program, this can be done by closing the listening socket (I think) , but I just installed an iptables rule.
- use the load balancer's API to offline the server(at the time it was F5).
I liked the first solution because it was load balancer agnostic, but required retries by the load balancer or client to avoid disruptions. Curious to hear what other people did.
https://landing.google.com/sre/book/chapters/load-balancing-...
Health checks are part of GCPs load balancer. https://cloud.google.com/compute/docs/load-balancing/health-...
However this approach need the binary to be started from socketmaster (as child process). I'm not sure what is the real cons tho, except the fact that it alter SIGHUP behavior - which is fine in this case imo.
[1] https://github.com/zimbatm/socketmaster
http://austinhanson.com/slinging-mud-graceful-reboots
Lol, no. Modern, professional software development uses a load balancer to manage a canary/red-black deployment strategy. Zero downtime deployments to a live server, outside of telecom/finance, is a sign of an amateurish team.