Once you separate the Guild process from all the user Session processes, by introducing Relays in the middle, a new problem seems to appear, and it's that now the routing between users needs to be handled differently.
Before, any Session could send a message to any other Session, and the Guild would work as the router.
Now, a Relay needs to discover which other Relay needs to receive a message. Or maybe all Relays do is to send the messages back to the Guild for routing?
Could you expand a bit more on that topic, please :)
Relays can forward things like "the user wants to join voice channel X" to the guild process and it'll process it (and fan out notifications to all the relays). This does place a scaling limit so we have to be careful about what things we forward to the guild to process.
Senior dev but Elixir newbie. I'll never have projects the scale of Discord, but listening to talks and podcasts I have some scalability questions that I think you could answer.
Some podcasters have mentioned their experiences with slowdowns and out-of-memory issues due to copying large pieces of data from one process to another to work on it. Their solution was to keep the large chunk of data in one process and instead pass in the functions needed to work on the data. Have you had a situation where you did something like that, for example with a list of members, or did you always handle that type of situation with ETS?
Some conventional wisdom I've seen is that you can get up to a cluster of about 40 nodes or so, then you start to have problems. Have you seen any issues with horizontal scaling at that scale or have you been able to handle everything by scaling up on your nodes?
Can you think of any other languages and VM's that could handle your particular scale, or is the BEAM the only option?
Yeah copying large amounts of data from one process to another (especially across nodes) is slow and expensive. We have to take care to not do that where possible. We've had issues from accidentally passing too much data to worker processes, but those are thankfully pretty rare because that is a pattern we try to avoid (having a single process per actor in the system is nicer for a variety of reasons).
Regarding cluster size: We haven't noticed too many problems with it, but that's because our communication is usually not a full mesh, but rather more like bipartite graphs (guilds talk to sessions, but not to other guilds, for instance).
Definitely other languages like C++, Rust, Go, or Java could solve similar problems, but we haven't had a reason to rewrite it all. Migrating only some types of actors in the systems to another language would mean not being able to use message passing / monitors and result in a lot of complexity because of it.
As one of the Midjourney volunteer moderators and a long time Discord user, it's been pretty cool to see how Discord has grown and changed to support Midjourney.
It seemed like the relationship was not great at the start, especially as things kept breaking with the growth, but then that changed and the relationship has felt really good with Discord listening to our feedback and helping fix issues we've noticed or implementing features for us to use.
7 comments
[ 2.4 ms ] story [ 34.2 ms ] threadBefore, any Session could send a message to any other Session, and the Guild would work as the router.
Now, a Relay needs to discover which other Relay needs to receive a message. Or maybe all Relays do is to send the messages back to the Guild for routing?
Could you expand a bit more on that topic, please :)
Some podcasters have mentioned their experiences with slowdowns and out-of-memory issues due to copying large pieces of data from one process to another to work on it. Their solution was to keep the large chunk of data in one process and instead pass in the functions needed to work on the data. Have you had a situation where you did something like that, for example with a list of members, or did you always handle that type of situation with ETS?
Some conventional wisdom I've seen is that you can get up to a cluster of about 40 nodes or so, then you start to have problems. Have you seen any issues with horizontal scaling at that scale or have you been able to handle everything by scaling up on your nodes?
Can you think of any other languages and VM's that could handle your particular scale, or is the BEAM the only option?
Regarding cluster size: We haven't noticed too many problems with it, but that's because our communication is usually not a full mesh, but rather more like bipartite graphs (guilds talk to sessions, but not to other guilds, for instance).
Definitely other languages like C++, Rust, Go, or Java could solve similar problems, but we haven't had a reason to rewrite it all. Migrating only some types of actors in the systems to another language would mean not being able to use message passing / monitors and result in a lot of complexity because of it.
It seemed like the relationship was not great at the start, especially as things kept breaking with the growth, but then that changed and the relationship has felt really good with Discord listening to our feedback and helping fix issues we've noticed or implementing features for us to use.