Show HN: Multiplayer Demo Built with Elixir (multiplayer.dev)
The current version of Realtime [1] is a Change Data Capture (CDC) server for a PostgreSQL database that broadcasts changes via WebSockets to authorized subscribers. It’s written in Elixir/Phoenix.
The server utilizes PostgreSQL’s logical replication functionality, which writes database changes to Write-Ahead Logging (WAL) segment files, and a replication slot, responsible for managing and retaining WAL files.
Database changes are polled from WAL by the server using PostgreSQL’s replication function pg_logical_slot_get_changes and changes converted to JSON objects using the wal2json [2] extension by setting it as the output plugin.
Security is enforced through two checks - each check ensures only authorized client subscribers are sent database changes. The first check validates a JWT that is sent by clients subscribing to database changes. This JWT must contain an existing database role and optional claims, both of which can be referenced in Row Level Security (RLS) policies. Every valid client subscription is then inserted into the realtime.subscription table with an assigned UUID, database role, and claims. The second check calls the realtime.apply_rls SQL function from Write Ahead Log Realtime Unified Security (WALRUS) utility lib [3]. This function takes the database changes, executes a prepared statement to verify if the database role and claims have SELECT permissions on the changes, and outputs an array of authorized UUIDs. Then, the server finds all the subscribers whose UUIDs are in that array and broadcasts the changes to them.
The next version of Supabase Realtime will offer three features: Broadcast, Presence, and Extensions.
Broadcast, our Pub/Sub offering, can be used to pass ephemeral data from client to client such as cursor movements. This runs on a distributed cluster of nodes built on top of Phoenix PubSub + Channels.
Presence, can be used for tracking online/offline users and their state. This is built into Phoenix, and uses replicated state across a cluster using an Observe-Remove-Set-Without-Tombstones (ORSWOT) CRDT [4] which prefers adds over removes when resolving conflicts.
Extensions, are a way for the community to add additional functionality to take advantage of the WebSocket infrastructure. We have converted the existing Change Data Capture system to an extension that supports connecting to multiple customer databases (multi-tenancy). Other possible extensions include listening to other databases like MySQL and getting stock market events server-side [5], then broadcasting them to connected clients.
This demo is built using a Supabase project, Supabase Realtime, and Next.js and deployed on 20 Fly [6] nodes located around the world. You can find an introduction and walkthrough of the demo here [5].
Supabase Realtime is entirely open source and you can find the demo code here [7]. Once we have stabilized the release we will add it to the self-hosted offering [8]. This demo is a way to highlight the upcoming features and gather feedback/ideas.
Feel free to ask me anything and let me know what you think!
[1] https://github.com/supabase/realtime
[2] https://github.com/eulerto/wal2json
[3] https://github.com/supabase/walrus
[4] https://gsd.di.uminho.pt/members/cbm/ps/delta-crdt-draft16ma...
[5] more details and full video in this blog post: https://supabase.com/blog/2022/04/01/supabase-realtime-with-... - Broadcast: Cursor movements and messages (floating bubbles next to cursors) sent directly to other players in the room. - Presence: Whenever someone joins the room, you'll be able to see that person represented as a colored circle in the upper right hand cover. When they leave, then their circle disappears. - Change Data Capture / Replication: When a player sends a message, that message is inserted into the database, and that change is then broadcast to all players in the room. You can see these message in the chat box in the lower right hand corner. - Latency: You can see how long it takes your client to send a message to the nearest node in the cluster and have the node respond with a message. This will vary depending on your internet. In particular when it comes to the database side you're going to want to limit individual realtime event writes. I suggest write to an append only log table and process the data in batches using a background worker. If you intend to insert every event one by one and index them various ways and have foreign keys to and from, inserting is going to incur a high cost. This can be mitigated for the most part with temporal partitioning, but even then it might be fine for you use case or analytic case to do background updates over an append only log and keep your appends quite fast. As for the network traffic and loading of the HTTP front end someone else on the team can maybe chime in on that. EDIT: Note this is very broad advice for any realtime app and not specific to this product. I think just being mindful about disk space and CPU usage will be helpful when setting up a database for logical replication because of changes being recorded to WAL files and having Realtime poll the database for changes. Our initial version of Realtime streams database changes to the server and those get broadcast to clients. However, the only security offered is checking whether or not a JWT is valid when client attempts to connect. The second and current version of Realtime, which I discussed in this post, checks database role and claims of subscriptions for every database change against RLS policies. This comes with a performance tradeoff as it prioritizes security. We are benchmarking this version in our new Supabase Realtime version and our team will try and optimize performance. > impacting performance with heavy use of multiplayer Realtime only needs a single connection to the database, and once a node gets the changes, it'll broadcast them to all other nodes in the cluster which will then be forwarded to all clients. This is highly scalable and nodes can be added in different regions experiencing higher loads. We're going to add in rate limiting to make sure that the cluster remains healthy, but those rate limits can be customized depending on the use case. It's also worth mentioning that "Presence" and "Channels" don't put any additional load on your database. You can broadcast messages (like the cursor movements in this demo) without it touching your database. For Broadcast, we can scale by adding additional nodes to the cluster but up to a point. The current architecture works because every node is aware of all other nodes in the cluster, but this strategy becomes untenable after a certain point. The good news is that we're nowhere near that point and we'll optimize and develop other strategies to circumvent this eventual limitation. We'll also need to carefully manage and process the inflow and outflow of messages on each node to make sure we're not delaying message delivery or using more memory than absolutely necessary. For Presence, it syncs the joins and leaves from node to node every 1.5 seconds (default) and there will be a point where this work will take longer than that interval. Fortunately, the underlying ORSWOT CRDT implementation can be refactored to make syncing more efficient. Like WoW in the early and middle years, the Open World feel is a bit of a fiction. WoW started with sharding akin to availability zones, but at ten million subscribers there’s only so much sharding you can do before people can’t meet each other. There were artificial choke points designed into both games that could be leveraged to offload some of the most intense interactions to separate hardware. Later WoW introduced phasing, which turned the map into a 4 dimensional space where actions were not observed by all observers (and map state could vary based on quest progress). The grouping system at that point had more situations where it could match you up with anyone in the same region not just the same availability zone, with or without moving you to a new hardware to do so. Everyone noted how brilliant it was going to be for scaling the system up, but I’ve found that the best scaling designs are bidirectional. Both of these games made a system where they could respond to post-content-release flash crowds, and also power more hardware down. The other thing Blizzard did that in a non-cloud world I found particularly clever: before a major content release they would upgrade hardware, migrate users to the new machines, use the old hardware for load shedding tasks, release the content, wait for traffic to subside, then decommission old hardware to get back to steady state. You get to test the new hardware before the Big Show, and rely on burned in hardware to get you through to the other side. Very good operational intelligence IMO. In Blizzard’s case some of the scaling tricks I mentioned earlier also improved the problem of users getting orphaned by declining subscriptions being spread out between too many AZs. In a pinch when traffic was low, you had access to the entire region (though I can’t say if that was controlled algorithmically or not). Previously the best they could do was a sort of traffic shaping by offering to move people from one server to another for free, where usually they charged a fee for doing so. That could take months to achieve load shedding. My suspicion is that at some points they had pools of hardware that belonged to the region and was parceled out between zones based on load. That allows you to absorb flash mobs better because those will be isolated to one zone instead of dependent on game state or time of day (after work on Friday everywhere with a low ping rate to that region, for instance). I have a YJS powered collaborative Svelte REPL I built using y-monaco and y-webrtc, all “serverless” on Vercel using CFWorkers to connect/verify users. Supabase already houses the private REPL loading/saving/sharing features. Native Supabase CRDTs could be perfect for the next version of the app! definitely. it's one of the primary use-cases we had in mind when developing this, and something we've wanted to offer for a long time. I'm happy to see it popping up early in the comments. We wouldn't offer our own CRDTs, but Realtime can be a nice transport layer for other CRDT implementations (which can then be serialised and persisted to your database) > YJS powered I'm also glad to see you're using Yjs - it's very cool. We hope that this implementation can be another Yjs Provider[0] if Kevin is onboard with that. Once that's implemented, you would be able to use it with all the same bindings (i.e. y-monaco). Re: ETS, I was just imagining something like Realtime but without Postgres, just using ETS. Most of the use cases I can think of for this type of thing don't require relational data, so I was thinking of just using ETS as a KV/document store to hold user and application state and then reacting to changes in to that the way you are here. I think you'd lose the benefits of relational queries and persistence but gain some function in overhead and decentralization. I'm moving to Elixir after spending a lot of time doing "serverless" and event-driven architectures, so now I'm spending a lot of time thinking about how to get the benefits of both worlds. Awesome initiative! > ETS as a KV/document store to hold user and application state and then reacting to changes in to that the way you are here This is actually pretty interesting. I can't speak to ETS but Mnesia has replication and you can expose the replication log using something like https://github.com/emqx/mria. I've only had a cursory look at this so I could be wrong about its capabilities but it would be an awesome extension to the new Realtime if possible. was this your path? I am heavily considering it, as I have been thinking about the BEAM for a couple years now but haven't dove in yet (coding is mostly hobby for me, though it does benefit my work sometimes) Did you have any trouble skipping Erlang and going straight to elixir? Simple Phoenix server that made requests to third party API endpoints. > Did you have any trouble skipping Erlang and going straight to elixir? I don't think I referred to any Erlang code and just picked up Elixir. Ultimately, you will have to pick up some Erlang but you can get away with just Elixir for a long time. Writing to the DB every 50ms for each cursor on the screen doesn't seem great for performance...or am I missing something? The new version adds "Channels", which are used for sending ephemeral/high-frequency events to other connected users. ```realtime-realtime-1 | Transport: :websocket
realtime-realtime-1 | Serializer: Phoenix.Socket.V1.JSONSerializer
realtime-realtime-1 | Parameters: %{"apikey" => "nokey", "vsn" => "1.0.0"}
realtime-realtime-1 | 2022-04-11 22:30:17.543 [warn] Ignoring unmatched topic "room:" in RealtimeWeb.UserSocket
realtime-realtime-1 | 2022-04-11 22:30:17.556 [warn] Ignoring unmatched topic "room:NAejZl5PKVbAwUsxCucM-" in RealtimeWeb.UserSocket
realtime-realtime-1 | 2022-04-11 22:31:11.144 [warn] Ignoring unmatched topic "room:" in RealtimeWeb.UserSocket
realtime-realtime-1 | 2022-04-11 22:31:11.166 [warn] Ignoring unmatched topic "room:jQCdG-h3NT8nFVk7BpCAc" in RealtimeWeb.UserSocket
```
Any idea what I might be missing? Can you say which of these new components will be open sourced? There are some other features (e.g. function hooks) that are also closed-source at the moment. Is Supabase heading for an “open core” model? Great observation! > I’ve learned a ton just from cruising around supabase GitHub. Glad to hear it! > Can you say which of these new components will be open sourced? All of these components are open source and licensed under Apache License v2.0. > There are some other features (e.g. function hooks) that are also closed-source at the moment. I actually worked on the initial implementation of function hooks. We've actually already open sourced both the client (see: https://github.com/supabase/supabase/tree/88bcef911669595428...) and the pg_net extension it requires (see: https://github.com/supabase/pg_net). I think we've yet to open source the SQL commands needed to create the schema, functions, etc. I'll talk to my team and we'll open source it. > Is Supabase heading for an “open core” model? I don't think so. We want to continue to open source our projects under either MIT (client libs) and Apache License v2.0 (server libs). And I will look more at the realtime libraries to see the new stuff. Thanks for all the open code!! Way to take the initiative! I think it was more of an oversight on our part than intentionally choosing to keep it closed. > And I will look more at the realtime libraries to see the new stuff. Awesome! We have a lot of updates and better documentation coming in the next couple of weeks so stay tuned. > Thanks for all the open code!! You're very welcome and it's the least we can do! Edit: Or, restrict it to pre-built messages. Love to hear it! > What would be the best way to build a multi-region cluster in Supabase? The easiest way we've found is to deploy the cluster on Fly.io. You can check out the docs here: https://fly.io/docs/getting-started/elixir.58 comments
[ 3.1 ms ] story [ 109 ms ] thread - Never
- Trust
- the
- Client
- Ever.
- EVER.