I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)
I think it's smart from a legal perspective, because the team members seem to partially be coming from companies acquired by Twitter.
So I guess, if you say "it's a Mastodon-clone", you cannot be accused of taking proprietary ideas from Twitter (this is just a guess, they know better).
But technically very interesting and refreshing to see. I really like their approach. It feels they are innovative.
From a legal perspective, it's against Mastodon's trademark policy: "Only use the Mastodon marks to accurately identify those goods or services that are built using the Mastodon software." https://joinmastodon.org/trademark
Yeah, I think this is just an ActivityPub server that supports the Mastodon extensions, right? I think we should embrace the fact that the federated world can be diverse, rather than just call everything "Mastodon"
Mastodon has it's own API. It basically offers a very limited ActivityPub API too, but it's own API is very different.
And it's a very slim ActivityPub inplementation. For example, I don't think you can do basic things like get an individual post in ActivityPub. This should be easy simple json-ld to get but it's just 404. https://www.w3.org/TR/activitypub/#retrieving-objects
It does have a bunch of stuff that isn't federated though, such as Like counts/collections. And of course it only implements the server-to-server (S2S) part of AP, not the client-to-server (C2S) part.
We call it a "Mastodon instance" because we implemented the entire Mastodon API (https://docs.joinmastodon.org/api/). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.
> We call it a "Mastodon instance" because we implemented the entire Mastodon API…
Except "Mastodon instance" means an instance of Mastodon, which is open source. Whether or not it was intended to be deceptive (I'd think a group of smart people would know better), this personally left a bad taste in my mouth.
>We spent nine person-months building our scalable Mastodon instance
They federated this brand new code in 9 months, and bluesky still hasn't released anything regarding federation. Don't keep your hopes up, it would kill their business model to let anyone run part of the network. People-driven networks are just not compatible with commercially driven ones, name one successful example.
"Originally, Twitter was one, monolithic application built with Ruby on Rails. But now, it's divided into about two hundred self-contained services that talk to each other. Each runs atop the JVM, with most written in Scala and some in Java and Clojure"[1]
So is Twitter not a Twitter instance? Like if it looks, walks and toots like a Mastodon, is it not a Mastodon instance?
That doesn't match the way people use the term though. Pleroma and Akkoma implement the Mastodon API but wouldn't be called Mastodon instances since they aren't running Mastodon.
the group involved here may want to be mindful of the Mastodon gGmbH trademarks. Using the Mastodon logo on redplanetlabs.com to pitch a reimplementation of ActivityPub might be seen as infringing.
Any trademark case is going to have to prove that a reasonable person would think this article is from Mastodon gGmbH, or is talking about their product "Mastodon".
The top of the page reads "Red Planet Labs", the title of the article is "How we reduced the cost of building Twitter at Twitter-scale by 100x" and the first line of the article is "We built a Twitter-scale Mastodon instance from scratch in only 10k lines of code."
No reasonable person is going to think that this article has anything to do with the official Mastodon software, so there's no trademark issue here.
Trademark law doesn’t work like that. You don’t get to license your trademark in the same way you get to license copyrighted works.
Otherwise, people could just say you can’t use their trademark in any document that says something negative about them, and then successfully sue the press and angry customers for complaining about them.
That's not the title of the article and also not what the article says. I would be really pissed if you editorialized the title of my article like that.
I'm happy to correct it if anyone suggests a better one. The intention is to find a neutral title that accurately reflects what the article itself is saying.
We've learned that when an article's original title generates complaints like https://news.ycombinator.com/item?id=37137317, the thread is likely to get derailed by shallow arguing about the title. It's in both the author's interest and the community's for us to nip that in the bud by (1) putting an accurate and neutral title at the top (preferably using representative language from the article itself), and (2) marking the title complaint offtopic since it no longer applies. These steps nudge the thread toward discussing the article's content rather than merely its title.
Alright, sounds reasonable. I think the problem here is that the author specifically says (in a sibling comment) that the point is not Mastodon and now it's in the title. Maybe they're fine with it though.
I'm no expert and definitely get things wrong - we only skim things and make a first crack at an attempt, and then rely on other people to refine it. If Nathan or someone else wants to suggest a more accurate and neutral title, we can do that - the goal is simply to clear the discussion space for something more interesting than title fever (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). But now I'm repeating myself!
Actually if you read the article you can see we tested way above Twitter-scale. We can easily run this instance at full Twitter-scale by just paying for more servers.
The point isn't the Mastodon instance, but rather that Rama enabled us to build it at scale with in a tiny amount of code and time.
Mastodon and Twitter don't do the same amount of work per post. Mastodon doesn't have a recommendation engine, they don't have an advertising engine, they don't scan every post for CSAM, there's no global search, etc. (Some of these things are good not to have, but they still drastically change the scope.)
Claiming to have enabled significant scaling of a Mastodon/ActivityPub-compatible instance is fine. Claiming to have replicated Twitter on the cheap is, from the post, not accurate.
That's why we're comparing it to the cost of Twitter's original consumer product. As a demonstration of Rama, we scoped this project to the entirety of Mastodon which is roughly equivalent to Twitter's original consumer product (actually, it's probably greater in scope with additional features like hashtag follows and more complex filter/mute capabilities).
All those use cases you listed absolutely can be implemented with Rama, and Rama's extreme cost benefits would apply to those as well.
The performance on the example Mastodon instance is very responsive - almost anywhere I clicked loaded nearly instantly. I created an account and the only thing I found missing was it doesn't implement full text search unless my user was tagged, but that might be a Mastodon specific item.
I think they have thought a lot about typical hard problems, such as having the timeline processing happen along side the pipeline, taking network / storage etc out of the picture. Nice work!
That is indeed an intentional part of Mastodon's design, which we tried to be faithful to as much as possible. We originally implemented search across all statuses and had to reimplement it when we realized Mastodon is a little different.
Well, we didn't start from anything as we implemented this completely from scratch. I believe Mastodon is much more widely used than those so it seemed like a better target for this.
Yeah about what I expect from a "we rebuilt twitter for cheap" post. There's no point to the comparisons with the Twitter codebase size/cost. It completely distracts from what is probably a perfectly fine project.
That's a fair criticism - this isn't an apples-to-apples comparison. What I find interesting about this is the cost of running the service. Being able to run a twitter-like thing on a hundred or so large aws instances is neat and I'm sure that many folks here dream of that kind of efficiency at their day jobs, but I'm more excited about how this scales down. Can you run a community of a thousand or so posters on a micro or nano instance for a few bucks a month or less? At that scale and cost, donations should easily be able to cover hosting fees and you would surely be able to deputize enough mods to keep things civil (for whatever definition of civil your instance lands on). Ads, monetization, personalization are non-issues (well, not major issues) at that scale.
I would not want to speak for raverbashing but I feel the same way: I actually can't tell if the bug is with soapbox or with your instance but clicking on the first link from your post practically locks up my browser due to every single Toot getting swapped out "at twitter scale"
If one clicks quick enough to jump to an actual post, it seems relatively static so it's hard to tell if the bots are deleting and recreating their posts or what. In true Xitter clone fashion, trying to view the Posts & replies from any one user is "sign in
Anyway, all of this is not to detract from your framework announcement as much as to have you consider that it's perfectly fine to label that instance as a load test, that's a fine thing, but calling it a legitimate instance seems to be a potential source of confusion
We did notice on a less powerful machines the browser getting overwhelmed with the rate of new content (even though we're only streaming 10/s instead of the full 3.5k/s actually happening on the backend). I don't know if the poor performance in this context is due to Soapbox, the browser, or just the hardware.
To get a better feeling of Rama's performance on your hardware, I suggest registering an account which will allow you to poke around the whole platform. It takes just a couple seconds to register and we don't send any emails.
> How is it possible that we’ve reduced the cost of building scalable applications by multiple orders of magnitude?
> You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doesn’t take that long to describe all the actions you can take on Mastodon and what those actions do. So the real question you should be asking is: given that software is entirely abstraction and automation, why does it take so long to build something you can describe in hours?
> At its core Rama is a coherent set of abstractions...
This conclusion is alarming to read from a company that's trying to sell a new platform. The vast majority of the work in building Twitter or Reddit is not about building a coherent set of abstractions, it's working with an often incoherent reality, dealing with a myriad of laws that describe, as if your web app were a human clerk at a post office, how to handle PII and credit cards and CSAM filters and audits and copyright claims and on and on...
I'm honestly shocked that the technical implementation of a simplified, coherent platform took a full 9 person-months. That shouldn't be the hard part. What I'd want to know as a prospective customer is how you handle exceptions to your beautiful, idealized architecture, when some foreign country requires that you only store comments posted by their citizens within their borders or something like that.
~~full text search doesn't appear to work... so it's possible they punted on one of the harder parts, which is fast efficient accurate fuzzy search, which moderation and a lot of those other harder things rely on.~~
eta: they say that had it but removed it because apparently it's not something mastodon supports. so I guess it is a pretty good high level implementation.
So the things that make it difficult are all things you shouldn’t be doing in the first place? Well that certainly helps.
You shouldn’t be handling PII/raw CC’s anyways (assuming FinTech is not your core business)
Secretly scanning your customers private messages against an illegal and immoral hash table from a pseudo-government entity? Are you law enforcement? No? Then fucking stop.
Copyright claims? Fuck ‘em. Only do what you are absolutely, positively, no way-out legally bound to do. No more no less. Require formal, written requests and comply in the maximum amount of time allowed.
Audits? What kind of audit? If they’re non-financial you’re probably doing something wrong.
Corporate squares have ruined the tech scene, and it’s time to resist.
Building Twitter/Mastodon *not at scale* isn't that hard and certainly doesn't take 200 person-years. Building it *at scale* is a completely different story. Remember the fail-whale? That was years of Twitter struggling to scale their product.
That said, as we described in the post our implementation of Mastodon is less code than Mastodon's official implementation. So not only is Rama orders of magnitude more efficient for building applications at scale, it's also much faster for building first versions of an application.
Well since you use clojure, you probably know that to have small codebase, people often pick clojure. Going from point A to point Z quickly is rarely a goal for startups, going through A.. B... C... quickly, is the goal. I am still looking through all this, but a thought of having to bet on some java api + hope and pray it will jump over all unknown hoops, hm.
Comparisons to twitter are unfair, twitter is not really technical gem or is it? It's pretty impressive to build it with 3 ppl in 3 months, but hmm also seems feasible using other tech, given all blueprints are out there.
Well, as mentioned in the post Instagram literally just built and released their own barebones Twitter clone this year, and it took them 25 person years. They were also able to leverage all their existing infrastructure powering similar products.
So I would not say it's remotely feasible to do this in less than one person-year with any other technology.
Hmmm, "Rama is programmed entirely with a Java API – no custom languages or DSLs" according to the landing page, but this sure looks like an embedded DSL for dataflow graphs to me - Expr and Ops everywhere. Odd angle to take.
I consider "DSL" as something that's its own language with it's own lexer and parser, like SQL. The Rama API is just Java – classes, interfaces, methods, etc. Everything you do in Rama, from defining indexes, performing queries, or writing dataflow ETLs, is done in Java.
Yep the original term DSL was for custom languages, the eventual introduction of using it for these kinds of literate APIs was done later. Using it in the original way unqualified is fine imo.
Whether you like it or not; internal DSLs became a thing with Ruby back in the day. And these days things like Kotlin also lend themselves pretty well to creating internal DSLs. Java is not ideal for this. Kotlin and Ruby have a few syntax features that make it very easy.
The difference is making an effort to expose the features of the library via a syntactically friendly way. Most Kotlin libraries indeed have nice APIs that are effectively mini DSLs.
If you need an example, Kotlin uses a nice internal DSL for HTML where you write things like
Div {
P {
+"Hello world"
}
}
This is valid Kotlin that happens to construct a dom tree. There is no magic going on here; just usage of a few syntax features of the language. Div and p here are normal functions that take a receiver block as the last parameter that receive an instance of the element they are creating. The + in front of the string is function with a header like this in the implementation of that element.
operator fun String.unaryPlus()
The same code in Java would be a lot messier because of the lack of syntactical support for this. You basically end up with a lot of method chaining, semicolumns and their convoluted lamda syntax. The article has a few examples that would look a lot cleaner if you'd rewrite them in Kotlin.
When someone makes a distinction that you don't immediately appreciate, maybe don't just dismiss it as splitting hairs, as if the world was a simple place.
You're missing the point. Rama is a generic platform that provides a new baseline for how expensive it is to build applications at scale. There's nothing about Rama specific to social networks. What we're showing is that Rama creates a new era in software engineering where the cost of building applications at scale is radically reduced. With Rama, anyone embarking on a new application today has a radically different economic outlook for the end-to-end cost of developing that application from prototype through large scale.
I think we provided a ton of substance backing up that claim, and we will provide even more next week when we release the build of Rama that anyone can use and its corresponding 100k words of documentation.
- "Depots" are event streams (for event sourced data repositories)
- ETL read one or more streams and project them to indexable read models...
- Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way.
- And you have query engine which splits a query into 1+ index sub-queries and then aggregates.
Am I missing something, this seems relatively standard event-sourced / CQRS-like architecture, but streamlined to avoid redundancy and reimplementation of common abstractions.
It would've helped if the terms were less obscure than "depots" and "PStates".
Systems that promise "free linear scaling" without qualifiers either withhold or have not analyzed/realized their bottlenecks yet. Say if there is eventual consistency maybe the "eventuality" becomes so long that the service fails at its purpose. Or the communication link bandwidth is exhausted between key business logic (mutation event generating) services, and so on.
The only systems that scale linearly are stateless systems. Mastodon is not stateless. And even stateless systems hit some bottlenecks eventually, as they exist and run in a scale-variant Universe.
So this claim by itself doesn't immediately impress me, just turns my red lights on, awaiting further investigation. But we can of course discuss why this claim is made and how is it supported. The article is long so I've not had the chance to read it entirely yet.
But we have X number of event streams mapped through Y number of ETLs to produce Z number of read model indices, in a shape that seems to form a highly interlinked DAG, which eventually loops back on itself in terms of message flow. Just the increased cross-chatter here as we introduce more features suggests non-linear scaling.
for example it can scale the way persistent data structures scale, which is to say "O(1) within target operational bounds" despite technically being log-n with high branch factor)
Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems.
You have the general model correct, but here are a few clarifications:
- PStates are partitioned, durable, replicated indexes that are represented as arbitrary combinations of data structures. A PState can be as simple an an integer per partition, or it can be complex like a map of lists of maps of sets. PStates allow you to shape your indexes to perfectly match your application's use cases.
- I wouldn't call Rama queries an "engine", as it's considerably more straightforward in how it works than something like SQL. The base query API is called "paths", which are an imperative way to concisely reach into one partition of one PState to fetch or aggregate values. There's also "query topologies" which are predefined, on-demand distributed computations that can fetch and aggregate data from many partitions of many PStates.
Thanks, I will read more soon! I'm curious... how do you resolve the "impedance mismatch" between some "canonical" models that business decisions are made, based upon, which need to be synchronous with the depots (and mutually synchronous with other models sharing fragments of the same data), and the eventually consistent read models, which have a more lax constraint on how up to date they are?
How do you ensure consistency here? How do you organize it in the data flow?
Say I update a user, because that user seems to still be there in the query result/indexes, but actually an event for this user being deleted has happened some time ago?
This can also happen I suppose of the depots run queries themselves on PState in order to determine if a certain event is valid at all or not, and how exactly to carry it out.
The impedance mismatches you're used to from using databases are gone because:
- You can finely tune your indexes to be exactly the optimal shape for your application (data structure). You can see this in our Mastodon implementation with the big variety of data structures we used for all the use cases.
- You're generally just using regular Java objects everywhere: appending to depots, during ETL processing, and stored in indexes.
How you coordinate data creation with view updates is a deeper topic, so I'll just summarize one of the basic mechanisms Rama provides for coordinating this. Depot appends can have an "ack level" that determines the conditions before Rama tells you that depot append has completed. The default level is "full ack" which includes all streaming topologies colocated with that depot fully processing that record. With this level, when the depot append completes you know that all associated indexes (PStates) have been updated.
There's also "append ack", which only waits for the depot append to be replicated on the depot, and "no ack", which is fire and forget. These all have their uses depending the specific needs of an application.
Thanks! So we can see these ACKs as "wait and synchronize" signals I suppose? However how can we ensure an "all or nothing" between all parties trying to ACK a conditions they're mutually dependent on? I.e. transactionality or atomicity?
Considering the length and amount of detail in this blog post, I understand why they would need another week to get the code ready (assuming there will be more docs)
Measuring words and loc is not a great way, imho, to share what you’re doing. In fact, I’d love a much shorter set of documentation now to under and this better. Long docs will probably make it less likely to read.
What one finds useful from a web application and what the web application actually is are usually two entirely different things.
I work in marketing automation, and I guess I have in one way or another my entire career. The clients who need to use the platform to communicate with their own clients over social networking may never touch our print delivery system, but that doesn't mean that print delivery doesn't exist or isn't important.
If you are unwilling to recreate the totality of the application in terms of functionality, then you are lying if you say that you have recreated it.
I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a change back to the user literally just after they made the change? How do you say "other views eventual-consistency is fine but for this view of this bit of info we need it updated now".
This write-up is very detailed but I couldn't find that explanation.
This is usually what I do. Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly.
One famous example of this going to far: Mac Mail app used to play a whoosh sound when your email is actually sent. They changed it to whoosh instantly no matter what. Given how often an email might fail to send or get delayed, this meant an actually useful indication of "great, your thing was sent, you can close your laptop now" was rendered useless.
Messaging apps often have a checkmark to indicate the message actually went to the server, and maybe another checkmark to indicate it was received on the other end. Maybe HN needs an icon indicating that your vote went through.
Yeah, it's easy enough that I was able to do it in the web inspector in a minute (artificial 1s network delay added): https://s11.gifyu.com/images/ScPMI.gif
I imagine you get some UUID back from your write, and effectively "block" until you see it committed to the event stream. The intent of such a system is certainly for the read-after-write latency to be not much longer than a traditional RDBMS. (This is roughly what the RDBMS is doing under the hood anyway.) Probably you can isolate latency-critical paths so they don't get stuck behind big stream processing jobs.
The advantage of the overall architecture is that nearly all application functionality (for something like a social network) can tolerate much higher latency than an RDBMS, so you really want to have architectural building blocks that let you actually use this headroom.
You write the update directly to the cache closest to the user and into the eventually consistent queue.
We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back then it was the server cache.
I have to say in my ~12 years as an active Redditor I can’t recall a time where I saw any real state issues, even with rapidly changing votes, etc. Bravo!? Now that we’re beyond the days of molten servers, I have to say its overall reliability in the face of massive spiky traffic is quite a feat.
Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.
It’s fast until it’s not. Making a post and then hitting reload and not seeing it can be very jarring for the user. Definitely something to think about.
What do you mean? Every post I do shows up instantly.
Reloading the page from scratch can be slow due to Soapbox doing a lot of stuff asynchronously from scratch (Soapbox is the open-source Mastodon interface that we're using to serve the frontend). https://soapbox.pub/
Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend.
Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.
That's part of designing Rama applications. Acking is only coordinated with colocated stream topologies – stream topologies consuming that depot from another module don't add any latency.
Internally Rama does a lot of dynamic auto-batching for both depot appends and stream ETLs to amortize the cost of things like replication. So additional colocated stream topologies don't necessarily add much cost (though that depends on how complex the topology is, of course).
In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the components it's describing seem to be equivalent to what he calls the Batch Layer and the Serving Layer in his book.
But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.
Rama codifies and integrates the concepts I described in my book, with the high level model being: indexes = function(data) and query = function(indexes). These correspond to "depots" (data) , "ETLs" (functions), "PStates" (indexes), and "queries" (functions).
Rama is not batch-based. That is, PStates are not materialized by recomputing from scratch. They're incrementally updated either with stream or microbatch processing. But PStates can be recomputed from the source data on depots if needed.
Forgive me if I’m misunderstanding things, but this seems quite similar to what Materialize and ReadySet do, but like “as a library”, because Rama doesn’t use a “separate” layer for the storage stuff. Is that correct-ish?
Hilariously, I went to edit the above comment, and HN was overloaded. Then it served me three or four 500's, AND it served me stale data in between
I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked.
From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model
(not knocking hacker news -- it's generally a very fast site, MUCH better than Reddit. Just that scaling beyond a single machine is difficult and full of gotchas )
StackOverflow used to run on a single (very beefy) machine also for a long time — databases make efficient use of vertical scaling, horizontal scaling is much harder.
Of course, specialist systems can often do much better.
When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding.
The way applications are built, and have been built since before I was born, is by combining together potentially dozens of narrow tools together: databases, computation systems, caches, monitoring tools, etc. There has never been a cohesive model capable of expressing arbitrary backends end-to-end, and every application built has to be twisted to fit onto the existing narrow pieces.
Rama is a lot more than just "event sourcing" and "materialized views". Those are two concepts at its foundation, but the real breakthrough is being that cohesive model capable of expressing diverse backends in their entirety. It took me more than five years of dedicated research to discover this model, and it was extremely difficult.
Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real.
But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs
(side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had a Rails app, a database, and RAID disks, and grew it from there. We had BigTable and perf reviews :-P )
Eventual consistency is probably one?
Can I specify that comment editing is correct and ACID, while likes/upvotes are eventually consistent? (No is a fine answer, these problems are hard)
I read through much of the doc, and don't see a mention of the word "consistency" at all, which seems like an oversight for something that is unifying what would be in a database with computation.
Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets materialized into any number of indexes.
You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot append with "full acking", which coordinates its response with the completion of colocated streaming ETLs. If you append at a lower level of acking, then you get eventual consistency on those PStates at the benefit of lower latency appends.
Microbatching is always eventually consistent as processing is asynchronous and uncoordinated to depot appends. Microbatching is higher thorughput than streaming and has simpler fault-tolerance semantics.
You'll be able to read a lot more about this when we release the docs next week.
I think many people are going to have problems programming with this consistency model, as they will with any that's different than a single machine. But that's basically "physics", so it's inevitable :)
But it seems like great work within the constraints -- look forward to learning more
I have indeed wondered why none of the cloud platforms have built more forward-looking tech like this -- instead it's copies of AWS and so forth
You have the option to track the latest update time and, during the minute immediately following this update, direct all reads to come from the leader. Additionally, you could oversee the replication lag among followers and block queries on any follower that lags more than a minute behind the leader.
For the client, it's feasible to retain the timestamp of its most recent write. In this way, the system can ensure that the replica responsible for any reads related to that user incorporates updates at minimum up to that recorded timestamp. If a replica isn't adequately current, the read can either be managed by another replica or the query can wait until the replica catches up. The timestamp might take the form of a logical timestamp, signifying the order of writes (e.g., log sequence number), or it could be based on the actual system clock, where synchronized clocks become vital.
When your replicas are spread across multiple datacenters—whether for user proximity or enhanced availability—there's an added layer of complexity. Requests requiring the leader's involvement must be directed to the datacenter housing the leader.
One strategy (somewhat common in lambda architectures) is to query both the long-term store and the in-flight operations, and blend the results. The in-flight stuff is both small and already in memory so it's pretty often trivially fast, even if blending the data is relatively complex.
That does limit you to operations/queries you can describe in this dual format, but pretty often that's fine. Or if you can relax read-after-write you can just ignore the in-flight stuff and read from the main store and then there are no (added) limitations.
In the year of our lord 2023, people are still launching immature products with "we built a clone of a tiny subset of Twitter" as their use case? Come on. Twitter is huge because they have to support a huge number of use cases. Using this proprietary framework won't magically make complex use cases go away.
As we mentioned in the article, Instagram just spent ~25 person-years building Threads which is a barebones clone of Twitter. Not only did we build our instance 30x faster than that, we have way more features like federation, hashtag follows, polls, DMs, global timelines, and more. And Instagram didn't start from scratch as Instagram/Meta already had infrastructure powering similar products.
It’s a massive ask, even if the platform was 100x better, for all developers to give up every programming language and database they’ve ever used to depend on a startups closed source platform for all functionality.
It’s hard enough trusting Google or Amazons cloud offerings won’t change.
It seems that’s what they’re proposing right? What am I missing?
We're actually not asking anyone to give up anything. First off, it has a simple integration API (which you'll be able to see the details of next week) that allows it to seamlessly integrate with any other backend tool (databases, monitoring systems, queues, etc.). So Rama can be incrementally introduced into any existing architecture.
Second, Rama has a pure Java API and is not a bespoke language. So no new language needs to be learned.
So Rama-powered apps need to be written in Java? Or will any JVM language work?
And the Rama core will remain closed-source? That part seems like the toughest sell of all, at a time when the vast majority of developer tooling and backends are open source or at the very least source-available.
Rama sounds interesting to me for my 'next big project', but I'd not even consider building it on top of a closed core. I think this is a pretty common sentiment in these circles.
I understand building an OSS business is not easy either. But perhaps there is some middle of the road that you can walk?
- A contractual obligation to open source all (now current) code a couple of years in the future?
- Or an almost-OSS license that makes life difficult for competing cloud providers, like https://www.mongodb.com/licensing/server-side-public-license... ?
Yep, not OSS licensed is a nope for us for core dependencies. I am aware many here like to sign away freedom (even future viability) for more productivity now (using some new fangled cloud thingy which might/will be gone in a few months to underpin your entire product), but we are not interested in that.
> Rama can be incrementally introduced into any existing architecture
Big if true (and if the opposite, of incrementally removing it also works). There have been similar platform efforts in past, such as https://news.ycombinator.com/item?id=20985429 . For that one, the "massive ask to give up every programming language and database they’ve ever used to depend on a startups closed source platform" seems like the biggest hindrance to adoption.
I can imagine this being really useful from the ground up. Because it looks like it wants to be the source of truth, with different views on the data.
It’s hard to imagine it for a complex legacy application without having lots of added complexity. It wants to be the unifying programming model for the application. It would seem like running with two RDMS sources of truth simultaneously.
It’s like the xkcd “there are 12 ways of doing X, let’s create a standard to unify them” now there are 13 ways
Summarizing, now edited down with some editorializing for clarity:
What is it? build web-scale reactive backends with an expressive java dataflow API. Instead of a database you develop your own custom app-specific indexes which are reactive, distributed and durable. It's like event sourcing and materialized views but integrated in a linearly scalable way.
> I cannot emphasize enough how much interacting with indexes as regular data structures instead of magical “data models” liberates backend programming
> It allows for true incremental reactivity from the backend up through the frontend. ... enable UI frameworks to be fully incremental instead of doing expensive diffs to find out what changed.
Ok, so in my mind I am positioning this against Materialized / differential dataflow, whose key primitive is a efficient streaming incremental join that works across very large relational tables. Materialized makes SQL reactive, Rama gives you a java dataflow DSL for developing purpose-built reactive database indexes.
How it works? 4 concepts: Depot, ETLs, PState, Query
Depots: "distributed, durable, and replicated logs of data." [Event streams?] "like Kafka except integrated" "All data coming into Rama comes in through depot appends."
ETLs: data arrives via depots, and is ETLed to PStates via "a Java dataflow API for coding topologies that is extremely expressive". "Most of the time spent programming Rama is spent making ETLs."
PStates seem like reactive data structures that are also durable/replicated, these are meant to supersede your database and indexes, letting you build custom purpose-built indexes that contain 100M elements:
> “partitioned states” are how data is indexed in Rama ... Unlike existing databases, which have rigid indexing models (e.g. “key-value”, “relational”, “column-oriented”, “document”, “graph”, etc.), PStates have a flexible indexing model. In fact, they have an indexing model already familiar to every programmer: data structures. A PState is an arbitrary combination of data structures. ... nested data structures can efficiently contain hundreds of millions of elements. For example, a “map of maps” is equivalent to a “document database”, and a “map of subindexed sorted maps” is equivalent to a “column-oriented database”. Any [composition] is valid – e.g. you can have a “map of lists of subindexed maps of lists of subindexed sets”.
Query: once you develop PStates to aggregate relevant data into a custom index of the right ... shape?, query seems sorta like GraphQL selectors over your custom index:
> Queries in Rama take advantage of the data structure orientation of PStates with a “path-based” API that allows you to concisely fetch and aggregate data from a single partition
> “query topologies” ... real-time distributed querying and aggregation over an arbitrary collection of PStates. These are the analogue of “predefined queries” in traditional databases, except programmed via the same Java API as used to program ETLs and far more capable.
Great question. There's actually two ways to look at this: what does it look like to run Rama in a unit test environment, and what does it look like to run a small-scale single-node Rama application in production?
For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in the post and this is what we're releasing next week.
For the latter, Rama can be run on a single node with each daemon and module being a separate process. We made it really easy to launch single-node Rama instances with just a couple commands with the "rama" script that comes with the release. That said, we haven't spent much time yet optimizing small-scale Rama deployments and there's likely things we can do to make it more efficient (e.g. combine the Conductor and Supervisor daemons into a single process).
Follow up question : do you see Rama as being a good fit for applications that /don't/ need Twitter scale? These have simpler requirements, but I feel the integration you propose could still have value there.
Yes, it's a better model for developing backends in general. Our comparison against Mastodon's official implementation demonstrates this, being at least 44% less code.
It's the ability to avoid the impedance mismatches which dominate existing tooling that makes such a difference. With existing databases, including RDBMS's, you have to twist your application to fit their data models. The existence of things like ORMs help, but they add their own layers of complexity.
With Rama, you mold your indexes to exactly match your application's needs. And you're always just working with objects represented however you want, whether appending data to depots, processing data in ETLs, or storing data in PStates.
That computation and storage are integrated and colocated is another way that Rama simplifies application development and deployment.
Something I'm immediately thinking about with this is change management and inertia at the early stages of a new, underdefined project. Less code is great, the big question is how such a system compares to the usual hack-and-slash method of getting a v1 up and running as you search for PMF from the perspectives of ops, cost, data migrations, rapid deployments, and so on. Presumably, the idea here is to start from the beginning with Rama, skipping over the usual "monolith fetches from RDBMS" happy paths, even for your basic prototype, this way you don't slip into a situation like Twitter did where that grew slowly into an unscalable monstrosity requiring a rewrite. So an article focused on the "easy" part that's required in the beginning of rapid change, as much as it's not as important as the "simple" part that shines later at scale, seems useful.
The basic operation Rama provides for evolving an application over time is "module update". This lets you update the code for an existing module, including adding new depots, PStates, and topologies.
Going by their claims, they are showing off their generalizable platform, Rama, by building an application on top of it. The application is an example, not the product. For example, someone implements a Todo app on their hot new javascript framework in 10 mins, your objection would be, "But it took you 2 years to make the framework, so actually it took you 2 years plus 10 mins". Why stop there? It also took many years to build the underlying language, networking layers, infrastructure, processors, materials etc etc. You have to draw the line at the point where the application specific code starts and the generalizable platform ends, no?
Exactly, why are so many people missing this point. It's not "we built a narrow, tedious framework for knocking off Twitter clones", it's "We built a platform that turns data processing on its head and look in a couple of months you can clone Twitter just imagine what YOU can do with this."
I see parallels though to Datomic, where they turned the database inside out, co-located the app logic and data and indexes, etc. There are a bunch of great videos on YT about Datomic by Rich Hickey & co, worth a watch and I think shine a light on the approach here, too.
I think they didn't do a good job making the point clear for people who just clicked the link without context. It starts off talking a lot about the Mastodon clone and then gradually starts talking about Rama as it goes.
The JVM took years to write. It took decades to develop the technology necessary to build a modern microcomputer. Before that, millennia to invent written language. And now that those platforms (including Rama) all exist, one can deliver a Mastodon server on top of them in about 9 man-months.
So if one was building a database, and then later building an app against their own database, they should not get to claim how much time the latter took alone? There are enough hyped bullshit in the world to call out, but I would not be so quick to dismiss this one, especially given that the author does know what he talks about (he used to work at twitter)
> The instance has 100M bots posting 3,500 times per second at 403 average fanout to demonstrate its scale.
Mastodon has to send messages to each instance with a recipient. That server can then fan out to all it's subscribers. The way this point is worded makes me think all the bits are on just a single instance, meaning all the fan out can be dealt with internally without having to do any server-to-server at all.
That is a fair comparison to Twitter, which is single instance. But it sounds like a much reduced ambition versus the task Mastodon has to do.
375 comments
[ 3.4 ms ] story [ 351 ms ] threadSo I guess, if you say "it's a Mastodon-clone", you cannot be accused of taking proprietary ideas from Twitter (this is just a guess, they know better).
But technically very interesting and refreshing to see. I really like their approach. It feels they are innovative.
And it's a very slim ActivityPub inplementation. For example, I don't think you can do basic things like get an individual post in ActivityPub. This should be easy simple json-ld to get but it's just 404. https://www.w3.org/TR/activitypub/#retrieving-objects
And either way, I think the source code to their Mastodonlike will not be usable since it will be running on their Rama server framework.
Mastodon is the name of a piece of software as well as an API, a website, etc.
Naming this stuff is hard but calling it a Mastodon instance would be more confusing.
Except "Mastodon instance" means an instance of Mastodon, which is open source. Whether or not it was intended to be deceptive (I'd think a group of smart people would know better), this personally left a bad taste in my mouth.
They federated this brand new code in 9 months, and bluesky still hasn't released anything regarding federation. Don't keep your hopes up, it would kill their business model to let anyone run part of the network. People-driven networks are just not compatible with commercially driven ones, name one successful example.
So is Twitter not a Twitter instance? Like if it looks, walks and toots like a Mastodon, is it not a Mastodon instance?
[1] https://www.wired.com/2013/09/the-second-coming-of-java/
https://joinmastodon.org/trademark
removed part about the mastodon subreddit since this is clearly not about the Mastodon software per se.
The top of the page reads "Red Planet Labs", the title of the article is "How we reduced the cost of building Twitter at Twitter-scale by 100x" and the first line of the article is "We built a Twitter-scale Mastodon instance from scratch in only 10k lines of code."
No reasonable person is going to think that this article has anything to do with the official Mastodon software, so there's no trademark issue here.
https://mastodon.redplanetlabs.com/timeline/local
and this from the gGmbH trademark policy:
> You may not use the Mastodon word mark, or any similar mark, in your domain name, unless you have written permission from Mastodon gGmbH.
Otherwise, people could just say you can’t use their trademark in any document that says something negative about them, and then successfully sue the press and angry customers for complaining about them.
https://wordpressfoundation.org/trademark-policy/
Article: "building Mastodon at sub-Twitter-scale"
We've learned that when an article's original title generates complaints like https://news.ycombinator.com/item?id=37137317, the thread is likely to get derailed by shallow arguing about the title. It's in both the author's interest and the community's for us to nip that in the bud by (1) putting an accurate and neutral title at the top (preferably using representative language from the article itself), and (2) marking the title complaint offtopic since it no longer applies. These steps nudge the thread toward discussing the article's content rather than merely its title.
The point isn't the Mastodon instance, but rather that Rama enabled us to build it at scale with in a tiny amount of code and time.
Claiming to have enabled significant scaling of a Mastodon/ActivityPub-compatible instance is fine. Claiming to have replicated Twitter on the cheap is, from the post, not accurate.
All those use cases you listed absolutely can be implemented with Rama, and Rama's extreme cost benefits would apply to those as well.
I think they have thought a lot about typical hard problems, such as having the timeline processing happen along side the pipeline, taking network / storage etc out of the picture. Nice work!
ctrl+f "monetization"
ctrl+f "moderation"
ctrl+f "existing infrastructure"
ctrl+f "personalization"
etc etc
Yeah about what I expect from a "we rebuilt twitter for cheap" post. There's no point to the comparisons with the Twitter codebase size/cost. It completely distracts from what is probably a perfectly fine project.
It's a JVM application with all state duplicated N times, so at least on the memory side it's likely going to be a resource hog.
Of course, for actual production use, there's probably a lot of things still, but this is a very nice works nonetheless
> The instance has 100M bots posting 3,500 times per second at 403 average fanout to demonstrate its scale.
If one clicks quick enough to jump to an actual post, it seems relatively static so it's hard to tell if the bots are deleting and recreating their posts or what. In true Xitter clone fashion, trying to view the Posts & replies from any one user is "sign in
Anyway, all of this is not to detract from your framework announcement as much as to have you consider that it's perfectly fine to label that instance as a load test, that's a fine thing, but calling it a legitimate instance seems to be a potential source of confusion
To get a better feeling of Rama's performance on your hardware, I suggest registering an account which will allow you to poke around the whole platform. It takes just a couple seconds to register and we don't send any emails.
> You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doesn’t take that long to describe all the actions you can take on Mastodon and what those actions do. So the real question you should be asking is: given that software is entirely abstraction and automation, why does it take so long to build something you can describe in hours?
> At its core Rama is a coherent set of abstractions...
This conclusion is alarming to read from a company that's trying to sell a new platform. The vast majority of the work in building Twitter or Reddit is not about building a coherent set of abstractions, it's working with an often incoherent reality, dealing with a myriad of laws that describe, as if your web app were a human clerk at a post office, how to handle PII and credit cards and CSAM filters and audits and copyright claims and on and on...
I'm honestly shocked that the technical implementation of a simplified, coherent platform took a full 9 person-months. That shouldn't be the hard part. What I'd want to know as a prospective customer is how you handle exceptions to your beautiful, idealized architecture, when some foreign country requires that you only store comments posted by their citizens within their borders or something like that.
eta: they say that had it but removed it because apparently it's not something mastodon supports. so I guess it is a pretty good high level implementation.
To be fair they developed this whole new platform to build this app with. I guess that's where the effort went.
> Our implementation is built on top of a new platform called Rama that we at Red Planet Labs have developed over the past 10 years.
You shouldn’t be handling PII/raw CC’s anyways (assuming FinTech is not your core business)
Secretly scanning your customers private messages against an illegal and immoral hash table from a pseudo-government entity? Are you law enforcement? No? Then fucking stop.
Copyright claims? Fuck ‘em. Only do what you are absolutely, positively, no way-out legally bound to do. No more no less. Require formal, written requests and comply in the maximum amount of time allowed.
Audits? What kind of audit? If they’re non-financial you’re probably doing something wrong.
Corporate squares have ruined the tech scene, and it’s time to resist.
That said, as we described in the post our implementation of Mastodon is less code than Mastodon's official implementation. So not only is Rama orders of magnitude more efficient for building applications at scale, it's also much faster for building first versions of an application.
Comparisons to twitter are unfair, twitter is not really technical gem or is it? It's pretty impressive to build it with 3 ppl in 3 months, but hmm also seems feasible using other tech, given all blueprints are out there.
So I would not say it's remotely feasible to do this in less than one person-year with any other technology.
https://www.washingtonpost.com/technology/2023/07/29/meta-th...
If you need an example, Kotlin uses a nice internal DSL for HTML where you write things like
This is valid Kotlin that happens to construct a dom tree. There is no magic going on here; just usage of a few syntax features of the language. Div and p here are normal functions that take a receiver block as the last parameter that receive an instance of the element they are creating. The + in front of the string is function with a header like this in the implementation of that element. The same code in Java would be a lot messier because of the lack of syntactical support for this. You basically end up with a lot of method chaining, semicolumns and their convoluted lamda syntax. The article has a few examples that would look a lot cleaner if you'd rewrite them in Kotlin.+ the time spent creating Rama, the platform that enables it.
Very dishonest leaving that out.
Bold of you to come to HN with the breathless hyperbolic marketing fluff that may work on Twitter...
- "Depots" are event streams (for event sourced data repositories)
- ETL read one or more streams and project them to indexable read models...
- Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way.
- And you have query engine which splits a query into 1+ index sub-queries and then aggregates.
Am I missing something, this seems relatively standard event-sourced / CQRS-like architecture, but streamlined to avoid redundancy and reimplementation of common abstractions.
It would've helped if the terms were less obscure than "depots" and "PStates".
The only systems that scale linearly are stateless systems. Mastodon is not stateless. And even stateless systems hit some bottlenecks eventually, as they exist and run in a scale-variant Universe.
So this claim by itself doesn't immediately impress me, just turns my red lights on, awaiting further investigation. But we can of course discuss why this claim is made and how is it supported. The article is long so I've not had the chance to read it entirely yet.
But we have X number of event streams mapped through Y number of ETLs to produce Z number of read model indices, in a shape that seems to form a highly interlinked DAG, which eventually loops back on itself in terms of message flow. Just the increased cross-chatter here as we introduce more features suggests non-linear scaling.
Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems.
You have the general model correct, but here are a few clarifications:
- PStates are partitioned, durable, replicated indexes that are represented as arbitrary combinations of data structures. A PState can be as simple an an integer per partition, or it can be complex like a map of lists of maps of sets. PStates allow you to shape your indexes to perfectly match your application's use cases.
- I wouldn't call Rama queries an "engine", as it's considerably more straightforward in how it works than something like SQL. The base query API is called "paths", which are an imperative way to concisely reach into one partition of one PState to fetch or aggregate values. There's also "query topologies" which are predefined, on-demand distributed computations that can fetch and aggregate data from many partitions of many PStates.
How do you ensure consistency here? How do you organize it in the data flow?
Say I update a user, because that user seems to still be there in the query result/indexes, but actually an event for this user being deleted has happened some time ago?
This can also happen I suppose of the depots run queries themselves on PState in order to determine if a certain event is valid at all or not, and how exactly to carry it out.
- You can finely tune your indexes to be exactly the optimal shape for your application (data structure). You can see this in our Mastodon implementation with the big variety of data structures we used for all the use cases. - You're generally just using regular Java objects everywhere: appending to depots, during ETL processing, and stored in indexes.
How you coordinate data creation with view updates is a deeper topic, so I'll just summarize one of the basic mechanisms Rama provides for coordinating this. Depot appends can have an "ack level" that determines the conditions before Rama tells you that depot append has completed. The default level is "full ack" which includes all streaming topologies colocated with that depot fully processing that record. With this level, when the depot append completes you know that all associated indexes (PStates) have been updated.
There's also "append ack", which only waits for the depot append to be replicated on the depot, and "no ack", which is fire and forget. These all have their uses depending the specific needs of an application.
FWIW, why hype at all? Why "We'll more in a week. Then more in two weeks." Show the code today!
I work in marketing automation, and I guess I have in one way or another my entire career. The clients who need to use the platform to communicate with their own clients over social networking may never touch our print delivery system, but that doesn't mean that print delivery doesn't exist or isn't important.
If you are unwilling to recreate the totality of the application in terms of functionality, then you are lying if you say that you have recreated it.
This write-up is very detailed but I couldn't find that explanation.
One famous example of this going to far: Mac Mail app used to play a whoosh sound when your email is actually sent. They changed it to whoosh instantly no matter what. Given how often an email might fail to send or get delayed, this meant an actually useful indication of "great, your thing was sent, you can close your laptop now" was rendered useless.
HN does this, and on slow days, about half of my upvotes don't go through.
Unfortunately the js used is very minimal. And if the upvote fails, there is no graphical indication that it failed to upvote.
The advantage of the overall architecture is that nearly all application functionality (for something like a social network) can tolerate much higher latency than an RDBMS, so you really want to have architectural building blocks that let you actually use this headroom.
You write the update directly to the cache closest to the user and into the eventually consistent queue.
We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back then it was the server cache.
I have to say in my ~12 years as an active Redditor I can’t recall a time where I saw any real state issues, even with rapidly changing votes, etc. Bravo!? Now that we’re beyond the days of molten servers, I have to say its overall reliability in the face of massive spiky traffic is quite a feat.
Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.
Reloading the page from scratch can be slow due to Soapbox doing a lot of stuff asynchronously from scratch (Soapbox is the open-source Mastodon interface that we're using to serve the frontend). https://soapbox.pub/
Rama's built-in telemetry provides the information you need to know when it's time to scale.
Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.
Internally Rama does a lot of dynamic auto-batching for both depot appends and stream ETLs to amortize the cost of things like replication. So additional colocated stream topologies don't necessarily add much cost (though that depends on how complex the topology is, of course).
But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.
Rama is not batch-based. That is, PStates are not materialized by recomputing from scratch. They're incrementally updated either with stream or microbatch processing. But PStates can be recomputed from the source data on depots if needed.
1. send event data to depot
2. trigger localized ETLs (or put it high-priority in queue) to recalculate just the impacted data into relevant PStates
3. await completion of aforementioned ETLs
4. run query from updated PStates
Maybe too heavy for an upvote, but very appropriate for a an important transaction like a purchase.
e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good
But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth
And yeah your question is an obvious good one, and the Reddit answer of "write through cache" ... is less than satisfying to me
I FREQUENTLY have the problem where I reload the page and Reddit shows me stale data. It's SUPER buggy.
---
Anyway I definitely look forward to hearing people try this and what their longer term impressions are !
I basically want to know what the tradeoffs are -- it sounds good, but there are always tradeoffs
So is the tradeoff "eventual consistency" ? What are the other tradeoffs?
I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked.
From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model
(not knocking hacker news -- it's generally a very fast site, MUCH better than Reddit. Just that scaling beyond a single machine is difficult and full of gotchas )
Of course, specialist systems can often do much better.
The way applications are built, and have been built since before I was born, is by combining together potentially dozens of narrow tools together: databases, computation systems, caches, monitoring tools, etc. There has never been a cohesive model capable of expressing arbitrary backends end-to-end, and every application built has to be twisted to fit onto the existing narrow pieces.
Rama is a lot more than just "event sourcing" and "materialized views". Those are two concepts at its foundation, but the real breakthrough is being that cohesive model capable of expressing diverse backends in their entirety. It took me more than five years of dedicated research to discover this model, and it was extremely difficult.
But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs
(side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had a Rails app, a database, and RAID disks, and grew it from there. We had BigTable and perf reviews :-P )
Eventual consistency is probably one?
Can I specify that comment editing is correct and ACID, while likes/upvotes are eventually consistent? (No is a fine answer, these problems are hard)
I read through much of the doc, and don't see a mention of the word "consistency" at all, which seems like an oversight for something that is unifying what would be in a database with computation.
You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot append with "full acking", which coordinates its response with the completion of colocated streaming ETLs. If you append at a lower level of acking, then you get eventual consistency on those PStates at the benefit of lower latency appends.
Microbatching is always eventually consistent as processing is asynchronous and uncoordinated to depot appends. Microbatching is higher thorughput than streaming and has simpler fault-tolerance semantics.
You'll be able to read a lot more about this when we release the docs next week.
I think many people are going to have problems programming with this consistency model, as they will with any that's different than a single machine. But that's basically "physics", so it's inevitable :)
But it seems like great work within the constraints -- look forward to learning more
I have indeed wondered why none of the cloud platforms have built more forward-looking tech like this -- instead it's copies of AWS and so forth
For the client, it's feasible to retain the timestamp of its most recent write. In this way, the system can ensure that the replica responsible for any reads related to that user incorporates updates at minimum up to that recorded timestamp. If a replica isn't adequately current, the read can either be managed by another replica or the query can wait until the replica catches up. The timestamp might take the form of a logical timestamp, signifying the order of writes (e.g., log sequence number), or it could be based on the actual system clock, where synchronized clocks become vital.
When your replicas are spread across multiple datacenters—whether for user proximity or enhanced availability—there's an added layer of complexity. Requests requiring the leader's involvement must be directed to the datacenter housing the leader.
https://avikdas.com/2020/04/13/scalability-concepts-read-aft...
https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_...
https://cloud.google.com/blog/products/databases/why-you-sho...
That does limit you to operations/queries you can describe in this dual format, but pretty often that's fine. Or if you can relax read-after-write you can just ignore the in-flight stuff and read from the main store and then there are no (added) limitations.
https://www.washingtonpost.com/technology/2023/07/29/meta-th...
> That's why we're comparing it to the cost of Twitter's original consumer product.
Plus, you cloned a preexisting architecture. FB wrote theirs from scratch. Not apples to apples. This is much easier.
It’s hard enough trusting Google or Amazons cloud offerings won’t change.
It seems that’s what they’re proposing right? What am I missing?
Second, Rama has a pure Java API and is not a bespoke language. So no new language needs to be learned.
Isn't Mastodon a Ruby On Rails application?
And the Rama core will remain closed-source? That part seems like the toughest sell of all, at a time when the vast majority of developer tooling and backends are open source or at the very least source-available.
We're keeping it closed-source for now.
Rama sounds interesting to me for my 'next big project', but I'd not even consider building it on top of a closed core. I think this is a pretty common sentiment in these circles.
I understand building an OSS business is not easy either. But perhaps there is some middle of the road that you can walk?
- A contractual obligation to open source all (now current) code a couple of years in the future? - Or an almost-OSS license that makes life difficult for competing cloud providers, like https://www.mongodb.com/licensing/server-side-public-license... ?
Big if true (and if the opposite, of incrementally removing it also works). There have been similar platform efforts in past, such as https://news.ycombinator.com/item?id=20985429 . For that one, the "massive ask to give up every programming language and database they’ve ever used to depend on a startups closed source platform" seems like the biggest hindrance to adoption.
It’s hard to imagine it for a complex legacy application without having lots of added complexity. It wants to be the unifying programming model for the application. It would seem like running with two RDMS sources of truth simultaneously.
It’s like the xkcd “there are 12 ways of doing X, let’s create a standard to unify them” now there are 13 ways
9, which is 3^2, and 27, which is 3^3. Or 900 is Yoda's age, and 27 which is the 27 club of musicians who committed suicide.
Move along, nothing to see here.
What is it? build web-scale reactive backends with an expressive java dataflow API. Instead of a database you develop your own custom app-specific indexes which are reactive, distributed and durable. It's like event sourcing and materialized views but integrated in a linearly scalable way.
> I cannot emphasize enough how much interacting with indexes as regular data structures instead of magical “data models” liberates backend programming
> It allows for true incremental reactivity from the backend up through the frontend. ... enable UI frameworks to be fully incremental instead of doing expensive diffs to find out what changed.
Ok, so in my mind I am positioning this against Materialized / differential dataflow, whose key primitive is a efficient streaming incremental join that works across very large relational tables. Materialized makes SQL reactive, Rama gives you a java dataflow DSL for developing purpose-built reactive database indexes.
How it works? 4 concepts: Depot, ETLs, PState, Query
Depots: "distributed, durable, and replicated logs of data." [Event streams?] "like Kafka except integrated" "All data coming into Rama comes in through depot appends."
ETLs: data arrives via depots, and is ETLed to PStates via "a Java dataflow API for coding topologies that is extremely expressive". "Most of the time spent programming Rama is spent making ETLs."
PStates seem like reactive data structures that are also durable/replicated, these are meant to supersede your database and indexes, letting you build custom purpose-built indexes that contain 100M elements:
> “partitioned states” are how data is indexed in Rama ... Unlike existing databases, which have rigid indexing models (e.g. “key-value”, “relational”, “column-oriented”, “document”, “graph”, etc.), PStates have a flexible indexing model. In fact, they have an indexing model already familiar to every programmer: data structures. A PState is an arbitrary combination of data structures. ... nested data structures can efficiently contain hundreds of millions of elements. For example, a “map of maps” is equivalent to a “document database”, and a “map of subindexed sorted maps” is equivalent to a “column-oriented database”. Any [composition] is valid – e.g. you can have a “map of lists of subindexed maps of lists of subindexed sets”.
Query: once you develop PStates to aggregate relevant data into a custom index of the right ... shape?, query seems sorta like GraphQL selectors over your custom index:
> Queries in Rama take advantage of the data structure orientation of PStates with a “path-based” API that allows you to concisely fetch and aggregate data from a single partition
> “query topologies” ... real-time distributed querying and aggregation over an arbitrary collection of PStates. These are the analogue of “predefined queries” in traditional databases, except programmed via the same Java API as used to program ETLs and far more capable.
For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in the post and this is what we're releasing next week.
For the latter, Rama can be run on a single node with each daemon and module being a separate process. We made it really easy to launch single-node Rama instances with just a couple commands with the "rama" script that comes with the release. That said, we haven't spent much time yet optimizing small-scale Rama deployments and there's likely things we can do to make it more efficient (e.g. combine the Conductor and Supervisor daemons into a single process).
It's the ability to avoid the impedance mismatches which dominate existing tooling that makes such a difference. With existing databases, including RDBMS's, you have to twist your application to fit their data models. The existence of things like ORMs help, but they add their own layers of complexity.
With Rama, you mold your indexes to exactly match your application's needs. And you're always just working with objects represented however you want, whether appending data to depots, processing data in ETLs, or storing data in PStates.
That computation and storage are integrated and colocated is another way that Rama simplifies application development and deployment.
[1] https://ghost.org/docs/install/
The basic operation Rama provides for evolving an application over time is "module update". This lets you update the code for an existing module, including adding new depots, PStates, and topologies.
Nono, you can't say that when later on you say it's built on top of Rama. You literally spent 10 years building the framework to even make this.
And yes, you built this in 10k lines of code but how many lines of code is Rama? This seems disingenuous.
I see parallels though to Datomic, where they turned the database inside out, co-located the app logic and data and indexes, etc. There are a bunch of great videos on YT about Datomic by Rich Hickey & co, worth a watch and I think shine a light on the approach here, too.
They had to invent the computer first, and before that they had to create a universe capable of sustaining both life and computers.
And you can't take the "twitter engine" out of twitter and build other apps with it. A lot of it is custom built to fit the twitter data model.
Unlike - it seems - Rama.
Mastodon has to send messages to each instance with a recipient. That server can then fan out to all it's subscribers. The way this point is worded makes me think all the bits are on just a single instance, meaning all the fan out can be dealt with internally without having to do any server-to-server at all.
That is a fair comparison to Twitter, which is single instance. But it sounds like a much reduced ambition versus the task Mastodon has to do.