I'd love to hear what a more lean-in approach might look like. Using something like Krustlet[1] to write an Erlang-specific kubelet would be a more work-with way to approach k8s that'd still allow for a wide variety of deploymemt models, including things like this use of a universal container that gets live updated.
Sometimes I feel that in modern distributed systems, we are very slowly and haphazardly recreating Erlang running on a IBM mainframe.
What if we could treat an entire datacenter as a unified whole, a la an IBM mainframe, and use an Erlang style actor system for cross-region communication/coordination?
I'd go further and assert that we're very slowly and haphazardly recreating an ad hoc, informally-specified, bug-ridden, slow implementation of half of Multics.
I have a vague feeling that this is fighting the framework… what is the usecase here? Do you often have very-long-lived websocket connections that need to be preserved over upgrades?
The “k8s way” here is to have your pods quiesce on rollout (old pods marked “unready” and removed from the Service), so existing connections are not torn down, but new connections get sent to the up-level pods. Then when all connections have drained from the down-level pods you terminate them. This would break down if you need to have say 1-hour websockets, but aren’t willing to wait 1h+ to roll out a new release. Is this a common requirement?
Interested to hear examples of cases where extremely long-lived connections are worth a lot of engineering pain to achieve.
At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) without any disruption to user traffic.
To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.
We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.
I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code.
Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.
The bit about migrating to other nodes doesn't flow from the builtin code loading functionality.
Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun.
[1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable.
We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.
This is so cool! I often wondered how possible would it be to keep a TCP connection alive during a server restart. How is the connection severed and the client not being aware of it works?
Probably a layer of indirection. One layer acting as the mailbox for sessions which then communicates with whatever services handles the connection to the user client. Effectively builds up a buffer until the service tree is restarted. Pub/sub abstractions basically. I could be wrong.
Some old game server processes would support a "copyover" function. They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket. Unsophisticated, but it works. It's completely transparent to the client because the connection is never actually closed. The worst the client might experience is some latency as their input gets buffered by the OS while the server process restarts.
Of course the connection would not persist over a physical server restart (or something like a pod being killed or whatever). I imagine it's possible to move a connection between nodes, though, with a fairly similar process.
>They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket
Nope. When a Process exits for whatever reason, the OS closes and releases all resources connected with all File/Socket descriptors for that Process.
Ah, yeah, "restart the process" was not the right description of what was happening in the cases I was thinking of - they'd exec themselves to hot reload code which I see is the first suggestion on the SA answer you linked.
Not quite. Terminating a process releases its filehandles, and if any sockets are now unreferenced, they are closed. But if there is a live filehandle in another process, the socket stays alive.
You can send socket file handles over UNIX domain sockets, so you could use this to export your list of sockets to persist over restart.
>For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.
Do you have a ballpark feel for the cost of just terminating connections? Is it mainly a tail latency thing where you might try to push a message down a socket that is currently reconnecting, and have to wait way longer than normal? Or a CPU/network overhead thing due to how many open connections you have, with a full restart causing a thundering herd of activity?
> Interested to hear examples of cases where extremely long-lived connections are worth a lot of engineering pain to achieve.
When I was at WhatsApp, I regularly observed connections from mobile phones (S60 mostly had the longest connection times, but all clients without a working platform push service will try to remain connected) in excess of 30 days. We weren't going against the framework to keep those though, we were all in on hot loading, and didn't do any container rigamarole until the move to Facebook. Keeping alive connections alive reduces work for client and server, and some networks are terrible and getting a connection takes many attempts, but often a working connection will continue to work; of course, some networks are terrible and timeout connections after 10 seconds of idle, but usually that's temporary.
If your 'long' connections are usually around a minute, like is common in http use cases, it's not too bad to keep old servers around while they drain, but if it's an hour or more, you'd need 2x the instances while you're rolling out, if you want to rollout in less than an hour... if you rollout quickly, and want to rollout a follow on, you're going to see even more. Depending on how you manage things, maybe the old and new instances can share machines, resource usage on the old should drop off quickly, but you might not be able to actually fit many instances on one machine, or resource allocation might not allow for it.
We would still have to kill connections in use sometimes; hot loading the BEAM could be possible, but it's not built for that and I haven't met anyone crazy enough to do it. Hot loading the FreeBSD kernel hasn't been done yet either (although, I have seen it for Linux). If there was a real need to keep TCP state while updating BEAM or the OS kernel, it would probably make more sense to build a way to transfer the states (tcp and corresponding application) among machines rather than to hot load all the things; it would probably be more approachable to develop that than to add hotloading to things that didn't plan for it. I don't know how WhatsApp manages today, I left sometime ago, and I avoided getting involved with the FB hosted servers; I know we had the ability to do hot loading, but it didn't really fit in the FB deployment model, and some people didn't like having multiple deployment methods.
28 comments
[ 2.9 ms ] story [ 21.2 ms ] thread[1] https://docs.krustlet.dev/topics/providers/
What if we could treat an entire datacenter as a unified whole, a la an IBM mainframe, and use an Erlang style actor system for cross-region communication/coordination?
Lots of interest.
The “k8s way” here is to have your pods quiesce on rollout (old pods marked “unready” and removed from the Service), so existing connections are not torn down, but new connections get sent to the up-level pods. Then when all connections have drained from the down-level pods you terminate them. This would break down if you need to have say 1-hour websockets, but aren’t willing to wait 1h+ to roll out a new release. Is this a common requirement?
Interested to hear examples of cases where extremely long-lived connections are worth a lot of engineering pain to achieve.
To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.
We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.
Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.
Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun.
[1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.
Some old game server processes would support a "copyover" function. They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket. Unsophisticated, but it works. It's completely transparent to the client because the connection is never actually closed. The worst the client might experience is some latency as their input gets buffered by the OS while the server process restarts.
Of course the connection would not persist over a physical server restart (or something like a pod being killed or whatever). I imagine it's possible to move a connection between nodes, though, with a fairly similar process.
Nope. When a Process exits for whatever reason, the OS closes and releases all resources connected with all File/Socket descriptors for that Process.
You will have to architect your System to explicitly accommodate this scenario. Some techniques here : https://stackoverflow.com/questions/55006657/can-i-allow-my-...
You can send socket file handles over UNIX domain sockets, so you could use this to export your list of sockets to persist over restart.
If you define a "Server" as a logical piece of functionality not directly mapped to a OS Process, then it can be done. See https://news.ycombinator.com/item?id=32067570
Is this based on Alex Snoeren's TCP Connection Migration draft ? - https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migr...
Do you have a ballpark feel for the cost of just terminating connections? Is it mainly a tail latency thing where you might try to push a message down a socket that is currently reconnecting, and have to wait way longer than normal? Or a CPU/network overhead thing due to how many open connections you have, with a full restart causing a thundering herd of activity?
When I was at WhatsApp, I regularly observed connections from mobile phones (S60 mostly had the longest connection times, but all clients without a working platform push service will try to remain connected) in excess of 30 days. We weren't going against the framework to keep those though, we were all in on hot loading, and didn't do any container rigamarole until the move to Facebook. Keeping alive connections alive reduces work for client and server, and some networks are terrible and getting a connection takes many attempts, but often a working connection will continue to work; of course, some networks are terrible and timeout connections after 10 seconds of idle, but usually that's temporary.
If your 'long' connections are usually around a minute, like is common in http use cases, it's not too bad to keep old servers around while they drain, but if it's an hour or more, you'd need 2x the instances while you're rolling out, if you want to rollout in less than an hour... if you rollout quickly, and want to rollout a follow on, you're going to see even more. Depending on how you manage things, maybe the old and new instances can share machines, resource usage on the old should drop off quickly, but you might not be able to actually fit many instances on one machine, or resource allocation might not allow for it.
We would still have to kill connections in use sometimes; hot loading the BEAM could be possible, but it's not built for that and I haven't met anyone crazy enough to do it. Hot loading the FreeBSD kernel hasn't been done yet either (although, I have seen it for Linux). If there was a real need to keep TCP state while updating BEAM or the OS kernel, it would probably make more sense to build a way to transfer the states (tcp and corresponding application) among machines rather than to hot load all the things; it would probably be more approachable to develop that than to add hotloading to things that didn't plan for it. I don't know how WhatsApp manages today, I left sometime ago, and I avoided getting involved with the FB hosted servers; I know we had the ability to do hot loading, but it didn't really fit in the FB deployment model, and some people didn't like having multiple deployment methods.