As the author, this sounds good to me! I'll probably even change the actual title to match. I originally was going to make it a question mark and the only reason I didn't is https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline... when I think the answer is probably "could probably be somewhat done" rather than "no".
Well this may be the first time that's ever happened :)
Betteridge antiexamples are always welcome. I once tried to joke that Mr. Betteridge had "retired" and promptly got corrected about his employment status (https://news.ycombinator.com/item?id=10393754).
I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.
Thanks for clarifying. I missed the max vs. average analysis because I was focused on the text. Still, as noted in the Rust code comment, the sample implementation doesn’t handle longer tweets.
URLs are shortened and the shortened size counts against the tweet size. The URL shortener could be a totally separate service that the core service never interacts with at all. Though I think in real twitter URLs may be partially expanded before tweets are sent to clients, so if you wanted to maintain that then the core service would need to interact with the URL shortener.
If it counted UTF-16 code units that would be dumb. It doesn't. The cutoff was deliberately set to keep the 140 character limit for CJK but increase it to 280 for the rest. And they did that based on observational data.
That size in bytes is based on the max size in UTF-8 and UTF-16. Codepoints below U+1100 are counted as one "character" by twitter and will need at most 2 bytes. Codepoints above it are counted as two "characters" by twitter and will need at most 4 bytes. Therefore 560 bytes, and it supports all languages.
Side note, this is more pessimistic than it needs to be, if you're willing to transcode. The larger codepoints fit into 20-21 bits, and the smaller ones fit into 12-13 bits.
I like this kind of exercise. One thing I am not seeing is analytics, logs and so forth that as I understand it are significant portions of Twitter's production cost story.
You need the blocks for the obvious for what we see because it is not necessarily obvious to everyone.
Over the last couple of months I've seen comments that summarise Twitter as a read-only service that doesn't have any real time posting requirements and similarly other comments that treat it as a write-only service with no real time read / fast search requirements.
Without _all_ the blocks even the simple surface level Twitter will have complexity people miss.
If it's this cheap to run you don't need analytics because you don't need to monetize it, and if it's this simple you don't need logs because it'll all work correctly the first time!
"You don't need to monetize it" who's going to fund your Twitter-as-a-charity? What happens when the free money goes away? Businesses have to pay the bills eventually one way or another, you need to plan for that in advance
If you're a small team running this and can actually deliver it with a single machine (or two), just charging a few bucks a month for verification should net enough money to run it and provide a decent living (and out of 300M users, there will be people who would pay).
Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.
On the other hand, Twitter does (or did) handle over 450 million monthly active users (based on stats websites), with a target for 315 monetizable daily active users (based on their earnings calls pre-privatization). Handling that amount of concurrency and beaming millions of tweets a day to home feeds and notifications is going to be logistically hard.
Is that 315 million monetizable DAUs? That sounds like a lot if the total is only 450 MAU. OTOH, 315k DAU seems like it wouldn't be enough to pay the bills.
There were some quarters with profit, some without; the past few years were mostly without IIRC.
They were targeting 315 mDAUs for Q4 2023, but in the final earnings it was only 238 mDAUs. Actual MAU stats weren't public iirc but some random stats sites seemed to say 450m global MAUs, which likely includes people with no ad preferences or who only view NSFW content (which can't be shown next to (most?) ads).
Posted this on a comment above but systems like Whatsapp likely sent an insane amount of data as well but used only 16 servers over 1.5 billion users at time of acquisition. Modern NICs can handle millions of requests a second - I still feel there is a lot of excess here.
Feels like the comparison is irrelevant. I'm guessing WhatsApp would have infrastructure challenges if all of their chats were group messages including the entirety of their user base, search, moderation, ranking, ads, etc. Isn't WhatsApp more comparable to only DMs?
> However, it does show how much real compute bloat these companies actually have
No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy.
A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work often goes to ancillary features that you never see as an end user.
Genuinely asking, why do you think Twitter needs 24 million vcpus to run?
This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their revenue on compute, maybe some of these features were not really necessary (but were just burning money)?
> This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users).
- 450m DAUs at the time of facebook acquisition [0]
- Twitter is not just DMs or Group Chat.
> It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently.
A fair comparision might have been Instagram: While Systrom did run a relatively lean eng org, they never had to monetize and got acquired before they got any bigger than ~50m?
WhatsApp used Erlang, a programming language designed for building concurrent, distributed systems, to help scale its messaging platform to support millions of users.
Erlang is particularly well-suited to building distributed systems because it was designed to handle failures at the process level rather than the hardware level. This means that if one part of a distributed system built with Erlang fails, the rest of the system can continue to operate without interruption.
This is critical for a messaging platform like WhatsApp, which needs to be able to handle millions of users sending messages simultaneously without experiencing any downtime. Additionally, Erlang's concurrency features allow it to support many thousands of simultaneous connections on a single machine, which is also important for a messaging platform that needs to be able to handle a large volume of traffic.
Presumably there's an entire data engineering / event processing pipeline that's being used to track user interactions at a fine grained level. These events are going to be aggregated and munged by various teams for things like business analytics, product / experiment feature analysis, ad analysis, as well as machine learning model feature development (just to name a few massive ones off the top of my head). Each of these will vary in their requirements of things like timeliness, how much compute is necessary to do their work, what systems / frameworks are used to do the aggregations or feature processing, and tolerance to failure.
> This is not apples to apples but Whatsapp
And yeah, whatsapp isn't even close to an apt comparison. It's a completely different business model with vastly different engineering requirements.
Is Twitter bloated? Perhaps, but it's probably driven by business reasons, not (just) because engineers just wanted to make a bunch of toys and promo projects (though this obviously always plays some role).
And for the most part, this herculean effort is wasted. Most people just want to see latest tweets from people they follow. Everything else is fluff to manipulate engagement metrics, pad resumes and attempt to turn twitter into something it's users never wanted.
Just guessing, but a lot of the resources are probably devoted to making money for the business, not padding resumes. Others have pointed it out, but showing tweets doesn't generate revenue without additional infrastructure.
Most people probably follow more people than they're capable of reading all the latest tweets of, so some sort of ranking/prioritisation makes total sense. And Twitter is ad funded, so they need to also show relevant ads where it makes sense/money.
Chat apps are mostly one on one interaction, it is much harder run an open platform where every user can potentially interact with every other user, not even talking about search and how complex it gets. If Twitter is bloated or not is a valid discussion, but comparison it to WhatsApp is not.
Ironically the one-to-many broadcasts are much easier to implement on a single box than as a scalable service spread across thousands of small container instances.
Because the actual product is not showing people tweets but to optimize who to show which ads based on their previous interactions with the site.
This is many orders of magnitude harder.
Being E2E encrypted, WhatsApp can’t do much with the content, so it is much closer to a naive bitshuffler than Twitter.
Twitter, while still not profitable (maybe it was in some recent quarters?) was much closer to it, having all the components necessary to form a reasonable ad business. For ads, analytics is critical, plus all the ad serving, plus it’s a totally different scale of compute being many to many rather than one to ~one.
1. Lots of batch jobs. Sometimes it's unclear how much value they produce / whether they're still used.
2. Twitter probably made a mistake early on in taking a fanout-on-write approach to populate feeds. This is super expensive and necessitates a lot of additional infrastructure. There is a good video about it here: https://www.youtube.com/watch?v=WEgCjwyXvwc
They did not spend half their revenue on compute. It’s more like 20-25% for running data enters/staff for DCs. Check their earnings report.
Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well.
The issue is storage, ads, and ML to name a few. For example, from 2015:
“ Our Hadoop filesystems host over 300PB of data on tens of thousands of servers. We scale HDFS by federating multiple namespaces.”
You can also see their hardware usage broken down by service as put in their blog.
Also Search (the article did says these wouldn’t fit to be fair but the discussion seems to be ignoring how much wouldn’t fit and why). Search is pretty expensive especially since to have it responsive you need the indexes to fit in memory—at least the Lucene variety, which at least in old YouTube videos Twitter used.
Because at that time, Whatsapp didn’t do all of the stuff that you need to do. No lawful intercept, no abuse detection/prevention, etc.
I don’t know enough about Twitter to assess their infrastructure, but I know that it easy to run lean until there’s a problem, and then you get trapped.
> This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users).
WhatsApp is mostly a bent pipe connecting user devices & relying on device storage. If WhatsApp had to implement Twitter functionality and business model (with same Engineers and stack), they'd need a lot more servers too. I'd hazard the number of servers would be in the same order of magnitude
Whatsapp is mostly message delivery from one person to a tiny group. It's absolutely trivial compared to Twitter.
In terms of what Twitter uses compute on, I'd guess analytics (Twitter measures "everything" for ad serving - go explore ads.twitter.com and analytics.twitter.com) and non-chronological timeline mixing both takes orders of magnitude more resources than the basic functionality.
> colo cost + total server cost/(3 year) => $18,471/year
Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Very often. I work with companies spending 10x as much on bad (bizarrely complex; indeed kubernetes, lambda, gateway, rds etc) setup and bad code on aws. Almost no traffic (b2b). Makes no sense at all.
It's a big meal. Those old-industry tycoons (oil/real estate/finance/alcohol/export/etc.) from gold rush era have too much money. There are a few places for money to land. Cloud seems to be the first wave, social media, messaging, big data, ai, smart everything. $200k-400k + $80k-$150k ranges are like hyenas army!
> spending more than this for dozens of kubernetes clusters on AWS before signing a single customer
Yup.
Cloud is 21st century Nickel & Diming.
Sure it sounds cheap, everything is priced in small sounding cents per unit.
But then it very quickly becomes a compounding vicious circle ... a dozen different cloud tools, each charged at cents per unit, those units often being measured in increments of hours....next thing you know is your cloud bill has as many zeros on the end of it as the number of cloud services you are using. ;-)
And that's before we start talking about the data egress costs.
With colo you can start off with two 1/4 rack spaces at two different sites for resilience. You can get quite a lot of bang for your buck in a 1/4 rack with today's kit.
> Until very recently, while money was still very cheap, the time overhead it would take to manage this just was not worth the cost savings.
You can also rent a whole server. There's not much difference in time in managing a VM in a cloud or a whole server you rent from someone. Depending on the vendor, maybe some more setup time, since low end hosts don't usually have great setup workflows, so maybe you need to fiddle with the ipmi console once or twice to get it started, but if you go with a higher tier provider, you can fully automate everything if that floats your boat. It's just bare metal rather than a VM, and typically much lower cost for sustained usage (if you're really scaling up signfigantly and down throughout the day, cloud costs can work out less, although some vendors offer bare metal by the hour, too)
We wasted 5-6 figures of salary dollars lovingly building those clusters and the automation surrounding them. We had blue-green zero downtime deploys, but no customers who would notice any downtime to begin with. I think the CTO just wanted k8s on his resume.
In one afternoon (at most), I could have written a script to deploy our demo with docker compose over ssh. Sure, docker compose won't scale forever, but their runway didn't last forever either.
Fair enough. To be honest, of the abstraction tools out there, Kubernetes is the one I'd hold off on as long as possible. Load balancing proxies, docker compose, auto-scaling, cloud databases, etc. are things I'd do relatively early, but not Kubernetes.
My friend mentioned this just before I published and I think that probably is the fastest largest thing you can get which would in some sense count as one machine. I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. Definitely more expensive than just networking a few standard machines though.
I also bet that mainframes have software solutions to a lot of the multi-tenancy and fault tolerance challenges with running systems on one machine that I mention.
> I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage.
Some commodity machines use external SAS to connect to more disk boxes. IMHO, there's not a real reason to keep images and tweets on the same server if you're going to need an external disk box anyway. Rather than getting a 4u server with a lot of disks and a 4u additional disk box, you may as well get 4u servers with a lot of disks each, use one for tweets and the other for images. Anyway, images are fairly easy to scale horizontally, there's not much simplicity gained by having them all in one host, like there is for tweets.
Yah like I say in the post, the exactly one machine thing is just for fun and as an illustration of how far vertical scaling can go, practically I'd definitely scale storage with many sharded smaller storage servers.
Incidentally, a lot of people have argued that the massive datacenters used by e.g. AWS are effectively single large ("warehouse-scale") computers. In a way, it seems that the mainframe has been reinvented.
I wouldn't really agree with this since those machines don't share address spaces or directly attached busses. Better to say it's a warehouse-scale "service" provided by many machines which are aggregated in various ways.
I wonder though.. could you emulate a 20k-core VM with 100 terabytes of RAM on a DC?
Ethernet is fast, you might be able to get in range of DRAM access with an RDMA setup. cache coherency would require some kind of crazy locking, but maybe you could do it with FPGAs attached to the RDMA controllers that implement something like Raft?
it'd be kind of pointless and crash the second any machine in the cluster dies, but kind of a cool idea.
it'd be fun to see what Task Manager would make of it if you could get it to last long enough to boot Windows.
I have fantasized about doing this as a startup, basically doing cache coherency protocols at the page table level with RDMA. There's some academic systems that do something like it but without the hypervisor part.
My joke fantasy startup is a cloud provider called one.computer where you just have a slider for the number of cores on your single instance, and it gives you a standard linux system that appears to have 10k cores. Most multithreaded software would absolutely trash the cache-coherency protocols and have poor performance, but it might be useful to easily turn embarrassingly parallel threaded map-reduces into multi-machine ones.
You absolutely can, but the speed of light is still going to be a limitting factor for RTT latencies, acquiring and releasing locks, obtaining data from memory, etc.
It's relatively easy to have it work slowly (reducing clocks to have a period higher than max latency), but becomes very hard to do at higher freqs.
Beowulf clusters can get you there to some extent, although you can always do better with specialized hardware and software (by then you're building a supercomputer...)
to me the line between machine and cluster is mostly about real-time and fate-sharing. multiple cores on a single machine can expect memory accesses to succeed, caches to be coherent, interrupts to trigger within a deadline, clocks not to skew, cores in a CPU not to drop out, etc.
in a cluster, communication isn't real-time. packets drop, fetches fail, clocks skew, machines reboot.
IPC is a gray area. the remote process might die, its threads might be preempted, etc. RTOSes make IPC work more like a single machine, while regular OSes make IPC more like a network call.
so to me, the datacenter-as-mainframe idea falls apart because you need massive amounts of software infrastructure to treat a cluster like a mainframe. you have to use Paxos or Raft for serializing operations, you have to shard data and handle failures, etc. etc.
but it's definitely getting closer, thanks to lots of distributed systems engineering.
It's a neat thought exercise, but wrong for so many reasons (there are probably like 100s). Some jump out: spam/abuse detection, ad relevance, open graph web previews, promoted tweets that don't appear in author timelines, blocks/mutes, etc. This program is what people think Twitter is, but there's a lot more to it.
I think every big internet service uses user-space networking where required, so that part isn't new.
I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list:
- spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution. Similar to ML timelines.
- ad relevance: Not a core feature if your costs are low enough. But see the ML estimates for how much throughput A100s have at dot producting ML embeddings.
- web previews: I'd do this by making it the client's responsibility. You'd lose trustworthiness though so users with hacked clients could make troll web previews, they can already do that for a site they control, but not a general site.
- blocks/mutes: Not a concern for the main timeline other than when using ML, when looking at replies will need to fetch blocks/mutes and filter. Whether this costs too much depends on how frequently people look at replies.
I'm fully aware that real Twitter has bajillions of features that I don't investigate, and you couldn't fit all of them on one machine. Many of them make up such a small fraction of load that you could still fit them. Others do indeed pose challenges, but ones similar to features I'd already discussed.
"web previews: I'd do this by making it the client's responsibility."
Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering (https://www.jwz.org/blog/2022/11/mastodon-stampede/)
For blocks/mutes, you have to account for retweets and quotes, it's just not a fun problem.
Shipping the product is much more difficult that what's in your post. It's not realistic at all, but it is fun to think about.
I do agree that some of this could be done better a decade later (like, using Rust for some things instead of Scala), but it was all considered. A single machine is a fun thing to think about, but not close to realistic. CPU time was not usually the concern in designing these systems.
I'll go ahead and quote that blog post because they block HN users using the referer header.
---
"Federation" now apparently means "DDoS yourself."
Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding.
The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons slows down.
Presumably each of those IPs is an instance, none of which share any caching infrastructure with each other, and this problem is going to scale with my number of followers (followers' instances).
This system is not a good system.
Update: Blocking the Mastodon user agent is a workaround for the DDoS. "(Mastodon|http\.rb)/". The side effect is that people on Mastodon who see links to my posts no longer get link previews, just the URL.
---
I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server. The comments suggest that he's hitting the database for every request instead of caching GET responses, but even with such a weird config a normal machine should be able to do over 10k/second without breaking a sweat.
While that may be funny, the number of Mastodon instances is growing rapidly, to the point where it will need to eventually be dealt with (not least because hosting on a Pi or having a badly optimized setup both happens in real life). But more to this example, it shows passing preview responsibility to end user clients is a far bigger problem. Eg not many would be able to handle the onslaught of being linked to from a highly viral tweet if previews weren't cached.
> I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server.
Mastodon is written on Ruby on Rails. That should really answer all your questions about the problem but if you're unfamiliar Ruby is slow compared to any compiled language, Rails is slow compared to near-every framework on the planet and it isn't written that well either.
> which is how many hard drives you can plug in to a non-mainframe machine for historical image storage.
You would be surprised. First off, SSDs are denser than hard drives now if you're willing to spend $$$.
Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-over-Ethernet, etc...
It's fairly easy to get several petabytes of fast storage directly managed by one box. The only limit is the total usable PCIe bandwidth of the CPUs, which for a current-gen EPYC 9004 series processors in a dual-socket configuration is something crazy like 512 GB/s. This vastly exceeds typical NIC speeds. You'd have to balance available bandwidth between multiple 400 Gbps NICs and disks to be able to saturate the system.
People really overestimate the data volume put out by a service like Twitter while simultaneously underestimating the bandwidth capability of a single server.
That's really cool! Each year of historical images I estimate at 2.8PB, so it would need to scale quite far to handle multiple years. How would you actually connect all those external drive chassis, is there some kind of chainable SAS or PCIe that can scale arbitrarily far? I consider NVMe-over-fabrics to be cheating and just using multiple machines and calling it one machine, but "one machine" is kinda an arbitrary stunt metric.
Stuff like [0] exists, allowing you to fan out a single server's PCIe to quite a few PCIe JBOD chassis. Considering that SSDs can get you ~1PB in 1U these days, you can get pretty far while still technically sticking with PCIe connectivity rather than NVMeoF.
Is an infiniband switch connected to a bunch of machines that expose NVMe targets really that different from a SAS expander connected to a bunch of JBOD enclosures? Only difference is that the former can scale beyond 256 drives per controller and fill an entire data center. You're still doing all the compute on one machine so I think it still counts.
It depends on how you think of "one machines". :) You can fit 1PB in 1U without something like NVMe-over-fabrics. So in a 4U unit gives you plenty of room.
We have Zen4c 128 Core with DDR5 now. We might get a 256 Core Zen6c with PCI-E 6.0 and DDR6 by 2026.
I really like these exercise of trying to shrink the amount of server needed, especially those on Web usage. And the mention of Mainframe. Which dont get enough credit for. I did something similar with Netflix 800Gbps's post. [1] Where they could serve every single user with less than 50 Racks by the end of this decade.
> People really overestimate the data volume put out by a service like Twitter while simultaneously underestimating the bandwidth capability of a single server.
It's outright comical. Above we have people thinking somehow amount of TLS connections single server can handle is a problem, in service where there would be hundreds of thousands lines of code to generate the content served over it, all while using numbers from what seems like 10+ years old server hardware
> I haven't looked into it, but I wouldn't be surprised if they could get around the trickiest constraint, which is how many hard drives you can plug in to a non-mainframe machine for historical image storage.
Netapp is at something > 300TB storage per node IIRC, but in any case it would make more sense to use some cloud service. AWS EFS and S3 don't have any (practically reachable) limit in size.
Because both are ridiculously slow to the point where they would be completely unusable for a service such as Twitter whose current latency is based off everything largely being in memory.
And Twitter already evaluated using the cloud for their core services and it was cost-prohibitive compared to on-premise.
Why not a single FPGA with 100Gbps ethernet or pcie with NVM attached?
Around $5K for the hardware and $5K for the traffic per month.
The software would be a bit trickier to write, but you now get 100x performance for the same price
I have seen a number of papers of people doing such a fun project as their PhD thesis. I'll try to find some more [1] examples with links on Google Scholar. Try searching for p4 programmable dataplane content address".
Let's spend multi-million dollars a year on a team of highly specialized FPGA engineers writing assembly and HDL so that we can save 5k a month. Feature velocity will be 100x slower as well, but at least our application is efficient.
I think that this may make sense for some applications, but I also think that if you can utilize software abstractions to improve developer efficiency, it reduces risk in the long run.
Those millions of dollars have already been spent. For example the P4 [1] language (a HDL language) and the Tofino 3 chip. It started out as FPGA (NetFPGA) to do programmable packet routing at linespeed. You now have the P4 language to define your packet routers with and generate your code. Ten years later we have 25 Tbps software defined packet routers.
That depends. If you would use our asynchronous runtime reconfigurable array called Morphle Logic [1] instead of FPGA (Field Programmable Gate Array), you could program this hardware Twitter in a 1000 hours and have it run at 50 Tbps.
You would not need to know much about hardware but you would get a thousandfold speedup of your software.
John Carmack tweeted something that made me noodle on this too:
>It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1]
Very interesting to see the idea worked out in more detail.
Keyword search and pagerank had working solutions decades ago.
Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw.
I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.
I don't underestimate the effort, I claim that it's not strictly necessary to have more than a couple people working on it for a minimum viable product, and less for a beta.
Do you think that the most successful web companies in the world with arguably the best people i.e. Amazon, Facebook, Instagram, TikTok, LinkedIn, Pinterest, Youtube, Netflix, Snapchat etc. have no idea what they are doing. That the highly complex, expensive and latency impacting recommendation systems could be replaced by trivial sorting.
Or maybe they do work, do translate to increased usage and do significantly impact revenue.
A lot of people don't realize that Dunning-Kruger can catch any of us unaware. It's easy to look at a problem, think about the surface level challenges you'd have building it and come to the conclusion that you could do it better or simpler.
Try changing your ordering on Twitter to chronological and see how much you miss.
It's not just ads, it means the set of people you follow becomes extremely critical for your experience in a way that makes it far less engaging.
That may be good or bad for you as a user depending on what you want, but for Twitter having most people stick to the ML augmented timeline is essential to keep you hooked.
which arrives at a browser running Tweak New Twitter as a browser extension, and that strips the response to just what the user actually wants to see. Effficiency!
Great, staple a few ML accelerators to your NIC. Nvidia sells them! You could build an entire supercomputer style setup 100% optimized for Twitter data movement and computation with COTS hardware IMO.
I strongly doubt that entire datacenters need to be used if and only if Twitter obsessively optimized for hardware usage efficiency over everything else. In reality they don't and make some pretty big compromises to actually get stuff built. Hardware is cheap, people are not.
a) No one has said that Twitter doesn't use ML accelerators.
b) No one has said Twitter operates entire data centres.
c) You need more than just NICs and ML accelerators to built a Twitter timeline. You need to rank the content, determine appropriate ads and combine them together. You can't do that in your network card.
Most projects I encounter these days instantly reach for kubernetes, containers and microservices or cloud functions.
I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical.
The thing most appealing about single server configs is the simplicity. The more simple a system easy, likely the more reliable and easy to understand.
The software thing most people are building these days can easily run lock stock and barrel on one machine.
I wrote a prototype for an in-memory message queue in Rust and ran it on the fastest EC2 instance I could and it was able to process nearly 8 million messages a second.
You could be forgiven for believing the only way to write software is is a giant kablooie of containers, microservices, cloud functions and kubernetes, because that's what the cloud vendors want you to do, and it's also because it seems to be the primary approach discussed. Every layer of such stuff add complexity, development, devops, maintenance, support, deployment, testing and (un)reliability. Single server systems can be dramatically mnore simple because you can trim is as close as possible down to just the code and the storage.
Kubernetes and containers are a means to service architecture; It enabled scalability but does not require it. You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server.
I don't even use containers - I aim primarily for simplicity and so far I have found I am able to build entire sophisticated systems without a single container. Containers I find make things much more complex.
Docker can be super simple. Like, if I want to run a Python service, that's just a few lines in a Dockerfile and a docker-compose.yml stub. Then I can trivially deploy that anywhere.
Observability in production is where APM solutions like Datadog, Elastic, and Sentry come in; you can go from just logging errors all the way up to continuously profiling your application and beaming log files to them to correlate with metrics and database query timings.
If you're just doing a simple application, Sentry really is the way to go, while Datadog and ELK are agent-based and more intended for complex setups and big enterprises (especially in their pricing structure/infra costs).
> You should still be containerizing your applications to ensure a consistent environment, even if you only throw it in a docker-compose file on your production server.
I'll say that this is a good point, especially because if you don't use containers or a similar solution (even things like shipping VM images, for all I care), you'll end up with environment drift, unless your application is a statically compiled executable with no system dependencies, like a JDK/.NET/Python/Ruby runtime or worse yet, an application server like Tomcat, all of which can have different versions. Worse yet, if you need to install packages on the system, for which you haven't pinned specific versions (e.g. needing something that's installed through apt/yum, rather than package.json or Gemfile, or requirements.txt and so on).
That said, even when you don't use containers, you can still benefit from some pretty nice suggestions that will help make the software you develop easier to manage and run: https://12factor.net/
I'd also suggest that you have a single mechanism for managing everything that you need to run, so if it's not containers and an orchestrator of some sort, at least write systemd services or an equivalent for every process or group of processes that should be running.
Disclaimer: I still think that containers are a good idea, just because of how much of a dumpsterfire managing different OSes, their packages, language runtimes, application dependencies, application executables, port mappings, application resource limits, configuration, logging and other aspects is. Kubernetes, perhaps a bit less so, although when it works, it gets the job done... passably. Then again, Docker Swarm to me felt better for smaller deployments (a better fit for what you want to do vs the resources you have), whereas Nomad was also pretty nice, even if HCL sadly doesn't use the Docker Compose specification.
When it comes to Java, everything could be used as a directory installation. Like you need JDK, maven and tomcat? Download and extract it somewhere. Modify your current PATH to include java and that's about it. You can build big tar.gz instead of OCI container which will work just fine.
So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.
And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.
After I've got some experience with Kubernetes, I'd reach for it always because it's very simple and easy to use. But that requires to go through some learning curve, for sure.
The best and unbeatable thing about containers is that there're plenty of ready ones. I have no idea how would I install postgres without apt. I guess I could download binaries (where?), put them somewhere, read docs, craft config file with data dir pointing to anotherwere and so on. That should be doable but that's time. I can docker run it in seconds and that's saved time. Another example is ingress-nginx + cert-manager. It would take hours if not days from me to craft set of scripts and configs to replicate thing which is available almost out of the box in k8s, well tested and just works.
> When it comes to Java, everything could be used as a directory installation. Like you need JDK, maven and tomcat? Download and extract it somewhere. Modify your current PATH to include java and that's about it. You can build big tar.gz instead of OCI container which will work just fine.
I've seen something similar in projects previously, it never worked all that well.
While the idea of shipping one archive with everything is pretty good, people don't want to include the full JDK and Tomcat installs with each software delivery, unlike with containers, where you get some benefit out of layer re-use when they haven't changed, while having the confidence that what you tested is what you'll ship. Shipping 100 app versions with the same JDK + Tomcat version will mean reused layers instead of 100 copies in the archives. And if you don't ship everything together, but merely suggest that release X should run on JDK version Y, the possibility of someone not following those instructions at least once approaches 100% with every next release.
Furthermore, Tomcat typically will need custom configuration for the app server, as well as configuration for the actual apps. This means that you'd need to store the configuration in a bunch of separate files and then apply (copy) it on top of the newly delivered version. But you can't really do that directly, so you'd need to use something like Meld to compare whether the newly shipped default configuration doesn't include something that your old custom configuration doesn't (e.g. something new in web.xml or server.xml). The same applies to something like cacerts within your JDK install, if you haven't bothered to set up custom files separately.
Worse yet, if people aren't really disciplined about all of this, you'll end up with configuration drift over time - where your dev environment will have configuration A, your test environment will have configuration B (which will sort of be like A), and staging or prod will have something else. You'll be able to ignore some of those differences until everything will go horribly wrong one day, or maybe you'll get degraded performance but without a clear reason for it.
> So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.
This is only viable/easy/not brittle when you have self-contained .jar files, which admittedly are pretty nice! Though if shipping JDK with each delivery isn't in the cards (for example, because of the space considerations), that's not safe either - I've seen performance degrade 10x because of a JDK patch release was different between two environments, all because of JDK being managed through the system packages.
Resource limits are generally doable, though Xms and Xmx lie to you, you'd need systemd slices or an equivalent for hard resource limits, which I haven't seen anyone seriously bother with, although they're at a risk of the entire server/VM becoming unresponsive should their process go rogue for whatever reason (e.g. CPU at 100%, which is arguably worse than OOM because of bad memory limits).
Ports are okay when you are actually in control of the software and nothing is hardcoded. Then again, another aspect is being able to run multiple versions of software at the same time (e.g. different MySQL/MariaDB releases for different services/projects on the same node), which most nix distributions are pretty bad at.
> And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.
I've seen this attempted, but it never worked properly - the codebases might not have been good, but those redeployments and integrating with Tomcat always lead to either memory leaks or odd cases of the app server breaking. That's why personally I actually enjoy the a...
So the goal is basically being able to do builds whilst running as few setup steps as possible.
Containers are a good common denominator because you essentially start with the OS, and then there's a file that automates installing further dependencies and building the artifact, which typically includes the important parts of the runtime environment.
- They're stupidly popular, so it basically nullifies the setup steps.
- Once setup, they by combinding both OS layers and App, they solve more of the problem and are therefore slightly more reliable.
- They're self-documenting as long as you understand bash, docker, and don't do weird shit like build an undocumented intermediary layer.
Infrastructure as Code does the same thing for the underlying infra layers and kuberenetes is one of the nicer / quicker implementations of this, but requires you have kubernetes available.
Together they largely solve the "works on my PC" problem.
Kubernetes just orchestrates containers. You can still run beefy machines and scale (if necessary) accordingly.
If anything, Kubernetes allows you to save cost by going with a scalable number of small, inexpensive, fully utilized machines, vs one large, expensive, underused one.
I would wager that the majority of users of k8s do so on a cloud where they could provision VMs of the proper size to begin with. The utilization argument is specious.
It's not. Utilization is a key metric in capacity planning of large scalable apps.
Capacity is based upon max utilization. A scaled web app is does not have constant utilization. The parent I was responding to suggested running on one large/face instance. Ok... if you're capacity planning, are you planning for peak rps or min rps? Obviously peak. Peak times are always a fraction of your total server uptime. This means one big/fast server would be underutilized most of the time.
How do you expect to dynamically vertically scale in cloud to fit demand while using a single server? Re-provision another server (either smaller or larger), redeploy all apps to the server, and then route traffic? Great, you're doing kubernetes job by hand.
> That doesn't really seem like an example, since the whole thing doesn't run one machine.
It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid.
There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to include the word "cloud".
> The database alone has multiple machines.
As I said, and the blog says ... there is only one writer. The other nodes are smaller read replicas.
Which again shows you don't need to go with the cloud buzzword-filled database services.
> It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid.
"100 million certs a year" is only like ~3 certs a second. Maybe twice in peak. That's not much. And you're doing same error, focusing on one core feature and ignoring everything around it
Kubernetes is useful if you have many teams working on things in parallel and you want them to deploy in similar ways to not have to reinvent the same wheel in 5 different ways by 5 different teams. If you don't have multiple teams, you don't need it.
Hah, exactly. It's not that you can't accomplish all the same things as k8s with your own bash scripts - it's that k8s exists to replace all your custom bash scripts!
> The thing most appealing about single server configs is the simplicity. The more simple a system easy, likely the more reliable and easy to understand.
Well you gotta have a backup strategy. I'm talking about the primary machine here, I assumed that would be obvious but maybe not. You build your failover strategy into your architecture - there's lots of ways to do it - I use Postgres so I would favor something based around log shipping.
And uptime is important, so you want to have that secondary running and ready, with a proxy in front of everything so you can switch as soon as you detect a failure. That's three hosts, plus your alerting has to be separate too, so that's four. Now, to orchestrate all this, we'll first get out Puppet...
If you're going from one machine to two, and you add an automatically failover mechansim, chances are your load switching mechanism is going to cause more downtime than just running from your single machine, and manually switching on failure (after being paged).
I think you should always plan for failures, but modern enterprise hardware is quite reliable. I would even posit that if you stood up a brand new physical server today, it has a good chance of beating AWS uptime (well, not the AWS dashboard numbers) over a one year period.
"hardware is quite reliable" is not a valid strategy. Hardware fails with some non-zero probability. You need to have a plan in place what to do if that happens, taking into account service disruption, backups etc.
Having a system in place that handles most of this gracefully (like kubernetes) is one way of having such a plan, there are others. Which one works best is dependent on your app, cost of downtime, your team that's tasked
with bringing everything back up in the middle if the night, etc.
People who leave details like this out when they say "kubernetes is complicated" just haven't seen the complexities of operating a service
well.
Keep in mind that the complexity of your distributed system is a liability itself. Most cloud or large company downtimes aren't due to hardware failure per-se but either operator error or unexpected interactions between components of their distributed system. You might just be trading one source of downtime for another.
> the complexities of operating a service well.
Keep in mind that in a lot of business applications, downtime isn't the end of the world and might be an accepted and priced-in "cost" of doing business. Operating it as you consider "well" would just cost them more with no benefit.
Because OP example is very simplistic and left on the table very important details, you would base 250M on a single machine? What about backups, obervability, how do you update that stack without bringing down everything ... Also this is napkin maths, this could be off by 10 or 100x which would change everything.
It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd of millions of users is completely different.
Four vms isn’t too bad to manage, I was the first dev at a startup that went prelaunch to acquisition from 2 App/DB servers in EC2 for a mobile app that hit #1 on both stores with a fairly high user tx rate and some ML stuff.
Heh, also praying that everything stays in the fast path. If a small portion of the workload uses a higher portion of machine resources then the moment an attacker figures it out they have a great way of DDOSing your service resources.
It’s worth noting that the author’s example doesn’t do anything like HTTP. It was purely an algorithmic benchmark.
Nobody should be looking at this and thinking that it’s realistic to actually serve a functional website at this scale on a single machine with actual real world requirements.
I never tried k3s, but what's wrong with kubeadm? I think that's literally two commands to run single server k8s: kubeadm init and kubectl taint something.
The only thing bad about single server kubernetes is that it'll eat like 1-2 GB of RAM by itself. When you whole server could be 256 MB, that's a lot of wasted RAM.
Projects are optimized to be developed by so-called ordinary developers.
We have python service which consumes gigabytes of RAM for quite simple task. I'm sure that I'd rewrite it with Rust to consume tens of megabytes of RAM at most. Probably even less.
But I don't have time for that, there are more important things to consider and gigabytes is not that bad. Especially when you have some hardware elasticity with cloud resources.
I think that if you can develop world-scale twitter which could run on a single computer, that's a great skill. But it's a rare skill. It's safer to develop world-scale twitter which will run on Kubernetes and will not require rare developer skills.
> The thing most appealing about single server configs is the simplicity.
In my experience this ended up with more complicated.
Those systems are typically developed by people who already left and are undocumented, and they become extremely difficult to figure out the config (packages, etc files... oh, where even the service files are located?) and almost impossible to reproduce.
It might be okay to leave it there, but when we need to modify or troubleshoot the system a nightmare begins...
Maybe I was just unlucky, but at least k8s configs are more organized and simpler than dealing with a whole custom configured Linux system.
I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way.
However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or other in place of the text of the quoted tweet itself but regardless a tweet takes up more than 280 unicode characters.
Also, nowhere in the article are hashtags mentioned. For a system like this to work you need some indexing of hashtags so you aren't doing a full scan of the entire tweet text of every tweet anytime someone decides to search for #YOLO. The system as proposed is missing a highly critical feature of the platform it purports to emulate. I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves.
Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets.
Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashtag-only search might fit, mainly because my impression is people doing hashtag searches are a small fraction of traffic nowadays so the main cost is disk, not sure though.
I did run the post by 5 ex-Twitter engineers and none of them said any of my estimates were super wrong, mainly just brought up additional features and things I didn't discuss (which I edited into the post before publishing). Still possible that they just didn't divulge or didn't know some number they knew that I estimated very wrong.
Sure, but if someone accuses your house of having issues, and you retort that you've had it inspected by professionals, a reply of "Hah! That's evidence, not proof!" is just a bit smarmy.
A few weeks ago there was in incident[0] in Jersey, where some people called fire fighters one evening because they could smell gas, the fire fighters didn’t find any leaks, and the building literally blew up the next morning. Experts make mistakes, and failing to understand that evidence != proof can literally kill people. Sometimes, making the distinction is smarmy; other times, it’s just being sensible.
Okay, but... we're spit balling database sizes. None of this is safety critical, or even in the general neighborhood of things where it's important enough to go and mathematically prove that our numbers are perfect.
This is critically wrong, and misses the point of the cliché entirely.
Absence of evidence, in your case via a clean building inspection, does not mean the building is safe. It just means the checklist of known items was considered and nothing bad found.
Ask a building inspector if their clean report proves nothing is wrong with the building.
They will be firm and quick to inform you that it’s not a warranty — anything not checked was not covered. Items not covered could still be significant problems.
That’s the whole point of the saying. Absence of evidence is not evidence of absence.
I don’t think that hashtags are a search only feature. In the posts themselves, the hashtags are clickable to view other tweets. I don’t think that qualifies as a search.
It does strike me as a feature you'd typically serve out of some sort of search index since if you had to build search, you'd essentially get indexing of hashtags "for free"
You are probably right and I am wrong. I just looked at a tweet and clicking the hashtag takes to the search page with that hashtag typed in. Probably implemented similarly behind the scenes. Though hashtag most likely does an exact match search instead of fuzzy searching for regular words and phrases.
I think the difficult part would be that tagging and indexing the relationship between a single tweet and all of its component hashtags (which you would then likely want metrics on to avoid needing to count indexes on, etc.) is where it would really start to inflate.
Another poster dug into some implementation details that I'm not going to go into. I think you could shoehorn it into an extremely large server alongside the rest of your project but then you're looking at processing overhead and capacity management around the indexes themselves starting to become a more substantial part of processing power. Consider that for each tweet you need to break out what hashtags are in it, create records, update indexes, and many times there's several hashtags in a given tweet.
When I last ran analytics on the firehose data (ca. 2015/16) I saw something like 20% of all tweets had 3 or more hashtags. I only remember this fact because I built a demo around doing that kind of analytics. That may have changed over time obviously, however without that kind of information we don't have a good guesstimate even of what storage and index management there looks like. I'd be curious if the former Twitter engineers you polled worked on the data storage side of things. Coming at it from the other end of things, I've met more than a few application engineers who genuinely have no clue how much work a DBA (or equivalent) does to get things stored and indexed well and responsively.
> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount.
While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.
This is indeed a problem, and one of the reasons to be careful about knowing you can fit things in one machine before taking that approach, because the moment you're forced to move to a distributed model it's rarely one machine to two but one machine to a dozen coupled with a major rearchitecturing effort at just the wrong moment.
It's often fine to start without a fully decoupled system (net present value of the time and money needed to scale out might be far too high), but you need to know whether or not it's likely to come and what to look for so you can start preparing in time.
Twitter's "linked" tweets seems to be strangely unattached from context.
What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database.
So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously.
I also heard that a method for spamming Twitter trending topics is to send tweets and delete them quickly.
My impression is that Twitter is big on real time processing. They definitely don't search the entire database for #YOLO tweets, instead they seem to be searching the almost-live stuff and some archived stuff(probably ranked and saved as noteworthy tweets or something).
Old stuff on twitter is weird. Tweets seem to eventually forget that You specifically liked a tweet, and will allow you to like a tweet again, and the like count will reflect you liking it twice.
Like count is almost surely just a locally incremented number, your like request will eventually get processed and will be dropped if it was already liked. It only has to be an eventually consistent value.
This is beyond optimistic updates (which twitter also has plenty of issues with - you'll reply to your tweet and it'll jump immediately to two replies, but on refresh just show your one).
> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform
I really wonder how much of a challenge this is and how much it occupies, not even talking about disk, but continuing the theoretical exercise in the linked URL, you can get 1U size servers with 2TB of RAM these days.
I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation.
Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of the water, but... maybe not by as much as people think...? I would not be surprised if a very performance-engineered version of Twitter could run on a few dozen racks full of beefy machines like this with HPC-grade super-fast fabric interconnects.
I have a strong sense that most large scale systems are way less efficient than what's possible. They trade development ease, modularity, and velocity for performance by using a lot of microservices, flabby but easy and flexible protocols (e.g. JSON over HTTP), slow dynamic languages, and layers of abstraction that could be integrated a lot more tightly.
Of course that may be a rational trade-off if velocity, flexibility, and labor costs and logistics matter more than hardware, power, or data center floor space costs.
I didn't get the impression that this would duplicate the entire functionality of Twitter, just what amounts to the MVP functionality. If you are only talking about the MVP it's at least somewhat plausible with a lot of careful engineering and highly efficient data manipulation.
I agree mostly. Where I differ in that I would argue that hashtags were THE thing that Twitter is most known for but that could be a perspective from having been on the platform for forever and a day and I recognize not everyone may make that same association anymore.
Biggest problem with this is the lack of considering analytics.twitter.com and ads.twitter.com. Twitter stores event data about everything that happens to a tweet, and lets you target ads with a lot of precision.
While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet views.
A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.
I think about this often. Specifically, how much bloat exists in the world because individuals in our society are forced to justify their existence on a daily basis.
Everyone has to be employed so it's better to keep adding more crap to products and make those products disposable in order to give people a job.
True. You'd also save a ton of less operations and engineering staff.
Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)
Compare the cost of operating multiple servers, on one hand, with the lost revenue from having weekly or monthly maintenance windows during which you just put up a Fail Whale page. Most people overestimate the latter by a huge margin.
That's fine if your service is really local - you can do it at night. Not really an option for a global site. Imagine if Twitter went down for a few hours every month. People are addicted to Twitter. It might be at a critical time for an entire country (e.g. the Queen dies). Even worse you can't guarantee how long the upgrade will take.
You'd definitely need at least two servers. But I think you could surely just have simple master/slave replication and switch between them.
Yeah, I was just responding to the "how about OS updates" part of the parent comment, for which scheduled downtime is a reasonable option. To protect your service from unscheduled downtime though, like a failing RAID array, you would need at least two servers.
Personally I wouldn't run a critical service on only one server, but two servers? Definitely doable. I actually have a service running on two servers in different DCs 700 miles apart. Zero downtime in 9 years. :)
Twitter is not serious enough that there cannot be a timeout.
With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes.
Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.
> With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes.
Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day?
> Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.
But for that mirroring, you need to replicate disk writes, databases, backups etc.. This additional load would easily bring the server to a point where a single server would no longer suffice, even an insanely spec'ed one.
> has an acceptable downtime of a few minutes a day?
We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.
There is a difference between open-ended, unscheduled downtime and a known 5 minute reboot window. If I can choose between an ad-free service that reboots at noon and a downtime-free service with ads, I would choose the ad-free service.
Who needs Twitter to be a service without any downtime?
Twitter actually used to have downtime during certain updates IIRC. The state department asked them to skip one such maintenance in service of some Iranian protests in 2009. I highly doubt that the era of US ops in twitter has ended, so downtime is probably a nonstarter.
> You'd also save a ton of less operations and engineering staff.
You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude.
EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be cheaper than buying a second server in any reasonable timeframe.
I was comparing the costs to building/operating on a single server to the real Twitter. Twitter is a massive, distributed, hugely complex system, which requires a large team to build, maintain, and operate. That costs $$$. Both for people and servers.
(You are right, servers are cheap compared to employee costs though.)
Not to mention that your business would end up being more profitable by avoiding GDPR fines. Most users in the EU would click "Reject all" anyway, so basically that code is not needed.
The hashtag index is not going to be any bigger than the tweets storage though, and may be significantly smaller, so this part is not if by an order of magnitude (even a binary one). Assuming something like a common SQL database is used for storage there would be a tags table (one row per unique tag, tag string plus a numeric identifier, indexed by both which bloats the size a bit but it'll still be small) and a link table (one row per tag in a message, containing the tag and message ids). Even if using 64-bit IDs because you don't want to fall over at 2 thousand million messages (or 4, if your DB supports unsigned types or you start at MININT instead of zero or one) then that structure is going to consume about 32 bytes per tag per message (plus some static overheads and a little more for non-leaf index pages of course). In theory this could be the same size as the messages table or even larger (if most messages contain many small tags), but in practise it is going to be significantly smaller.
Yes, this would be big enough to need specifically factoring into a real implementation design. But it would not be big enough to invalidate the proposed idea so I understand leaving it off, at least initially, to simplify the thought experiment.
Similarly to support a message responding to, or quoting, a single other you only need one or two NULLable ID references, 16 bytes per message, which will likely be dwarfed by the message text in the average case. Given it likely makes sense to use something like SQL Server's compression options for data like this the average load imposed will be much smaller than 16 bytes/message.
We are fiddling, fairly insignificantly, measurable but to massively, with constants a & b in O(a+bN) here, so the storage problem is still essentially of order O(N) [where N is total length of the stored messages].
> I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves.
I'd probably go as far to say that the indexes _generally_ at twitter could be larger than the tweets
Hashtags aren't like user accounts, no - they're strings that are part of a tweet. In theory, a separate data structure shouldn't be needed since you can just search the full text of tweets, but in practice, I don't know how that scales for the number of all-time tweets.
> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount.
I want to add another concept that may impact, considerably, the storage, which is "threads". I'm not sure what is the percentage of threads/tweets but what I consider an important factor is that threads do not have a maximum number of characters.
This is absolutely the sort of thing I wish more developers did - and I think the good ones already do. Most of what you find in blogs will work just fine at 1 request per second (OMG! 1M Hits!!) or 10 requests per second (and I think someone did post their “how I scaled to 10 million hits per month” blog to Hacker News once which sounds impressive until you do the math) but when you get into thousands of requests per second you really do need to understand the network stack, the different storage tiers, your choice of algorithms, how to interact with CDNs, etc. a lot more then any blog will tell you.
When interviewing developers I always ask them what is the largest public web site they ever worked on and then probe about performance issues they encountered and how they resolved them in order gague how far along they are in their skill development.
I would never plan to run a production service on a single server just because coordinating changes in the active dataset among two or more production servers often changes your design significantly, and you want to plan for that because the consumer grade hardware we all use has a nasty habit of not working after power cycles (which still tends to be the most strain a system goes through, even in a world of SSD storage).
How much bandwidth does Twitter use for images and videos? Less than 1.4Tb/s globally? If so, we could probably fit that onto a second machine. We can currently serve over 700Gb/s from a dual-socket Milan based server[1]. I'm still waiting for hardware, but assuming there are no new bottlenecks, that should directly scale up to 1.4Tb/s with Genoa and ConnectX-7, given the IO pathways are all at least twice the bandwidth of the previous generation.
There are storage size issues (like how big is their long tail; quite large I'd imagine), but its a fun thing to think about.
That’s a not so simple calculation as comparing the raw request process time isn’t the complete picture. Spam bot content must be persistently stored, indexed, archived, etc so the long term cost is much more than the one off POST to create the entity.
You’d also have to quantify the improved user experience from seeing less spam v.s. inflated ad revenue for garbage views / content.
This is never going to happen. If I want to indicate it's correct I'd write Tbit/s or Tbyte/s. Otherwise its a coin flip if TB and Tb has been used correctly.
I like /s for bytes and ps for bits. 100Mbps = 100 million bits per second. 100MB/s = 100 million bytes per second. (The capitalization is important of course. I tried writing some examples where it's not preserved and it's too weird. I tend to not use capital letters on things like Slack, but for bytes, you just have to. The difference between milli and Mega can also be important, but since nobody talks about the negative powers with bandwidth, you are probably OK if your shift key breaks.)
No need, for those you can display a "Twitter is experiencing issues" or something similar. That will encourage the user to return to the main app and enjoying infinite scroll.
HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests.
For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then.
And that's for HTTP without encryption or ssl handshake. Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ].
I agree most HTTP server benchmarks are highly misleading in that way, and mention in my post how disappointed I am at the lack of good benchmarks. I also agree that typical HTTP servers would fall over at much lower new connection loads.
I'm talking about a hypothetical HTTPS server that used optimized kernel-bypass networking. Here's a kernel-bypass HTTP server benchmarked doing 50k new connections per core second while re-using nginx code: https://github.com/F-Stack/f-stack. But I don't know of anyone who's done something similar with HTTPS support.
"Quant uses the warpcore zero-copy userspace UDP/IP stack, which in addition to running on on top of the standard Socket API has support for the netmap fast packet I/O framework, as well as the Particle and RIOT IoT stacks. Quant hence supports traditional POSIX platforms (Linux, MacOS, FreeBSD, etc.) as well as embedded systems."
TLS handling would dominate your performance, kernel bypassing would not help here unless you would also do TLS NIC offloading, you still need to process new TLS sessions from OP example and they would dominate your http processing time (excluding application business logic processing).
I once built a quick and dirty load testing tool for a public facing service we built. The tool was pretty simple - something like https://github.com/bojand/ghz but with traffic and data patterns closer to what we expected to see in the real world. We used argo-workflows to generate scale.
One thing which we noticed was that there was a considerable difference in performance characteristics based on how we parallelized the load testing tool (multiple threads, multiple processes, multiple kubernetes pods, pods forced to be distributed across nodes).
I think that when you run non-distrubuted load tests you benefit from bunch of cool things which happen with http2 and Linux (multiplexing, resource sharing etc) which might make applications seem much faster than they would be in the real world.
And I would say real life Twitter involves mostly cell phone use where we see companies like Google try to push HTTP/3 to deal with head of line issues on lossy connections. Serving at the millions of hits per day on lossy networks is going to leave you with massive numbers of connections that have been abandoned but you don't know it yet. Or connections that are behaving like they are tar pitted and running at bits per second.
Vertical scaling doesn't have to be a single machine. You can do a lot with a half dozen machines split for different responsibilities, like we did in the 90's and 00's. Database, web servers, reverse proxy.
> Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ].
I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.
I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.
All dynamic content, all hitting data storage. There are no simulated clients, this is all real traffic from real clients, a lot of requests do writes, some do only reads.
I guess that the biggest chunk of a slowdown from TLS comes due to group operations alone. So wouldn't it be practical to configure TLS for session resumption and limit the number of handshakes per second it could do?
In the coming years we will probably see a lot of complicated microservice architectures be replaced by well-designed and optimized Rust (and modern C++) monoliths that use simple replication to scale horizontally.
Replication and simple never belong in the same sentence. DNS which is one of the simplest replication systems I know of has its own complex failure modes.
CockroachDB is nice to use but every database has complexity you have to deal with.
Here's one I ran into recently: if a range has only 1 of 3 replicas online then it will stop accepting traffic for that range until it has 3 replicas again.
(for the folks at home, "range" is a technical term for 512 bit slice of the data - CRDB replicates at the range level)
So, in some code I wrote, I had account for not only 1) the whole DB being unavailable but also 2) just one replica being unavailable (they're different failure modes that say different things about the health of the system).
It's a good behavior! Good for durability. But I had to do some work to deal with it, spend an hour coming up with a solution, etc. There are databases that work at Twitter scale but no there are no silver bullets among those that do. You need full time engineers to manage the complexity and keep it online, or else it could cost the company shitloads of money - I've seen websites of similar scale where a two-hour outage cost them $20 million.
You're right, no progress has ever been made in software, no new ideas are better than any old ideas, and it's fads all the way down. The only difference between software today and software in 1980 is that today's software is hip and software from 1980 is square.
I understand the frustration with flavor of the week "best practices" and the constant churn of frameworks and ideas, but software engineering as a practice IS moving forward. The difficulty is separating the good ideas (CI/CD, for example) from the trends (TDD all the things all the time) ahead of time.
You don't even need Rust or highly optimised code. Just moving the existing code from "vCPUs" and networked storage to real CPUs with direct-attach NVME storage will be enough for most purposes. (btw you can do that now, just get yourself a beefy server at OVH/Hetzner and play around with it)
I see a lot of comments here assuming that this proves something about Twitter being inefficient. Before you jump to conclusions, take a look at the author’s code: https://github.com/trishume/twitterperf
Notably absent are things like serving HTTP, not to even mention HTTPS. This was a fun exercise in algorithms, I/O, and benchmarking. It wasn’t actually imitating anything that resembles actual Twitter or even a usable website.
Which I think I'm perfectly clear about in the blog post. The post is mostly about napkin math systems analysis, which does cover HTTP and HTTPS.
I'm now somewhat confident I could implement this if I tried, but it would take many years, the prototype and math is to check whether there's anything that would stop me if I tried and be a fun blog post about what systems are capable of.
I've worked on a team building a system to handle millions of messages per second per machine, and spending weeks doing math and building performance prototypes like this is exactly what we did before we built it for real.
Not to the extreme of fitting everything into one machine but I have explorer the idea of separate stateless workload into its own machine.
However, the stateless workload can still operate in a read-only manner if the stateful component failed.
I run an email forwarding service[1], and one of challenge is how can I ensure the email forwarding still work even if my primary database failed.
And I come up with a design that the app boot up, and load entire routing data from my postgres into its memory data structure, and persisted to local storage. So if postgres datbase failed, as long as I have an instance of those app(which I can run as many as I can), the system continue to work for existing customer.
The app use listen/notify to load new data from postgres into its memory.
Not exactly the same concept as the artcile, but the idea is that we try to design the system in a way where it can operate fully on a single machine. Another cool thing is that it easiser to test this, instead of loading data from Postgres, it can load from config files, so essentially the core biz logic is isolated into a single machine.
Essentially IO is expensive except within a datacenter but even in a data center, you can do a lot of loop iterations in a hot loop in the time it takes to ask a server for something.
There is a whitepaper which talks about the raw throughput and performance of single core systems outperforming scalable systems. These should be required reading of those developing distributed systems.
I think one of the under-estimated interesting points of twitter as a business is that this is the core. Yes, Twitter is 140 characters, it's got "300m users" which is probably 5m real heavy users. So yes, you could do a lot of "140 characters, a few tweets per person, few million users" on very little hardware. But that's why Twitters a shit business!
How much RAM did your advertising network need? Becuase that is what makes twitter a business! How are you building your advertiser profiles? Where are you accounting for fast roll out of a Snapchat/Instagram/BeReal/Tiktok equivalent? Oh look, your 140 characters just turned into a few hundreds megs of video that you're going to transcode 16 different ways for Qos. Ruh Roh!
How are your 1,000 engineers going to push their code to production on one machine?
Almost always the answer to "do more work" or "buy more machines" is "buy more machines".
All I'm saying is I'd change it to "Toy twitter on one machine" not Production.
The author claimed early on, and very clearly that this was a fun exercise of thought and engineering rather than saying “Look this is how Twitter should be run”. After all this is Hacker News. Such exercises, and engaging other hackers to pick something out of there is how we progress(and get our tickles). So, may be instead think about how one could tackle the advertising/indexing needs in a similar fashion(could it be done in just another server? 5 more servers?)..
Yeah I completely get that, but I think a lot of hacker news tends to think of companies as the sum of their engineering resources, rather than what they really are. Which is weird, because it's hosted by YC which is meant to be the polar opposite of that. The point of my comment was that the decision making that lead to Twitter's design actually make sense when you understand the business model behind them. It's not good enough to just run a webpage.
If it actually takes you a single machine to run this, you don't really need an advertising network to fund it. Out of 5M users (let alone the theoretical 300M) there will be enough people who'd be happy to pay for verification or an exclusive badge on their profile.
> How are your 1,000 engineers going to push their code to production on one machine?
That might actually be the reason why Twitter barely keeps afloat. 1k engineers for a product that's already built and hasn't fundamentally changed nor evolved in years makes me wonder what business value those engineers actually provide.
I remember Stack Overflow running on a single Windows Server box and mocking fellow LAMP developers with their propensity towards having dozens of VMs to same effect.
499 comments
[ 6.3 ms ] story [ 367 ms ] threadBetteridge antiexamples are always welcome. I once tried to joke that Mr. Betteridge had "retired" and promptly got corrected about his employment status (https://news.ycombinator.com/item?id=10393754).
Edit: Still a nice writeup!
280 * 4 = 1120, not 560.
If it counted UTF-16 code units that would be dumb. It doesn't. The cutoff was deliberately set to keep the 140 character limit for CJK but increase it to 280 for the rest. And they did that based on observational data.
https://cdn.cms-twdigitalassets.com/content/dam/blog-twitter...
https://blog.twitter.com/en_us/topics/product/2017/Giving-yo...
Side note, this is more pessimistic than it needs to be, if you're willing to transcode. The larger codepoints fit into 20-21 bits, and the smaller ones fit into 12-13 bits.
Over the last couple of months I've seen comments that summarise Twitter as a read-only service that doesn't have any real time posting requirements and similarly other comments that treat it as a write-only service with no real time read / fast search requirements.
Without _all_ the blocks even the simple surface level Twitter will have complexity people miss.
They were targeting 315 mDAUs for Q4 2023, but in the final earnings it was only 238 mDAUs. Actual MAU stats weren't public iirc but some random stats sites seemed to say 450m global MAUs, which likely includes people with no ad preferences or who only view NSFW content (which can't be shown next to (most?) ads).
https://www.forbes.com/sites/johnkoetsier/2022/11/14/twitter...
No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy.
A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work often goes to ancillary features that you never see as an end user.
This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their revenue on compute, maybe some of these features were not really necessary (but were just burning money)?
- 450m DAUs at the time of facebook acquisition [0]
- Twitter is not just DMs or Group Chat.
> It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently.
A fair comparision might have been Instagram: While Systrom did run a relatively lean eng org, they never had to monetize and got acquired before they got any bigger than ~50m?
[0] https://www.sequoiacap.com/article/four-numbers-that-explain...
Erlang is particularly well-suited to building distributed systems because it was designed to handle failures at the process level rather than the hardware level. This means that if one part of a distributed system built with Erlang fails, the rest of the system can continue to operate without interruption.
This is critical for a messaging platform like WhatsApp, which needs to be able to handle millions of users sending messages simultaneously without experiencing any downtime. Additionally, Erlang's concurrency features allow it to support many thousands of simultaneous connections on a single machine, which is also important for a messaging platform that needs to be able to handle a large volume of traffic.
> This is not apples to apples but Whatsapp
And yeah, whatsapp isn't even close to an apt comparison. It's a completely different business model with vastly different engineering requirements.
Is Twitter bloated? Perhaps, but it's probably driven by business reasons, not (just) because engineers just wanted to make a bunch of toys and promo projects (though this obviously always plays some role).
Twitter, while still not profitable (maybe it was in some recent quarters?) was much closer to it, having all the components necessary to form a reasonable ad business. For ads, analytics is critical, plus all the ad serving, plus it’s a totally different scale of compute being many to many rather than one to ~one.
1. Lots of batch jobs. Sometimes it's unclear how much value they produce / whether they're still used.
2. Twitter probably made a mistake early on in taking a fanout-on-write approach to populate feeds. This is super expensive and necessitates a lot of additional infrastructure. There is a good video about it here: https://www.youtube.com/watch?v=WEgCjwyXvwc
Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well.
The issue is storage, ads, and ML to name a few. For example, from 2015:
“ Our Hadoop filesystems host over 300PB of data on tens of thousands of servers. We scale HDFS by federating multiple namespaces.”
You can also see their hardware usage broken down by service as put in their blog.
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
https://blog.twitter.com/engineering/en_us/a/2015/hadoop-fil....
I don’t know enough about Twitter to assess their infrastructure, but I know that it easy to run lean until there’s a problem, and then you get trapped.
WhatsApp is mostly a bent pipe connecting user devices & relying on device storage. If WhatsApp had to implement Twitter functionality and business model (with same Engineers and stack), they'd need a lot more servers too. I'd hazard the number of servers would be in the same order of magnitude
In terms of what Twitter uses compute on, I'd guess analytics (Twitter measures "everything" for ad serving - go explore ads.twitter.com and analytics.twitter.com) and non-chronological timeline mixing both takes orders of magnitude more resources than the basic functionality.
Meanwhile the company I just left was spending more than this for dozens of kubernetes clusters on AWS before signing a single customer. Sometimes I wonder what I'm still doing in this industry.
Yup.
Cloud is 21st century Nickel & Diming.
Sure it sounds cheap, everything is priced in small sounding cents per unit.
But then it very quickly becomes a compounding vicious circle ... a dozen different cloud tools, each charged at cents per unit, those units often being measured in increments of hours....next thing you know is your cloud bill has as many zeros on the end of it as the number of cloud services you are using. ;-)
And that's before we start talking about the data egress costs.
With colo you can start off with two 1/4 rack spaces at two different sites for resilience. You can get quite a lot of bang for your buck in a 1/4 rack with today's kit.
Until very recently, while money was still very cheap, the time overhead it would take to manage this just was not worth the cost savings.
Even with the market falling out from under VC, I think it still is a good tradeoff for many shops.
You can also rent a whole server. There's not much difference in time in managing a VM in a cloud or a whole server you rent from someone. Depending on the vendor, maybe some more setup time, since low end hosts don't usually have great setup workflows, so maybe you need to fiddle with the ipmi console once or twice to get it started, but if you go with a higher tier provider, you can fully automate everything if that floats your boat. It's just bare metal rather than a VM, and typically much lower cost for sustained usage (if you're really scaling up signfigantly and down throughout the day, cloud costs can work out less, although some vendors offer bare metal by the hour, too)
In one afternoon (at most), I could have written a script to deploy our demo with docker compose over ssh. Sure, docker compose won't scale forever, but their runway didn't last forever either.
You can get up to $100k and it's a big reason many startups go in that direction.
Also $20k is nothing when you factor in developer time etc.
You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory.
I'm not saying this is a good idea, but it seems better than what the OP proposes.
I also bet that mainframes have software solutions to a lot of the multi-tenancy and fault tolerance challenges with running systems on one machine that I mention.
Some commodity machines use external SAS to connect to more disk boxes. IMHO, there's not a real reason to keep images and tweets on the same server if you're going to need an external disk box anyway. Rather than getting a 4u server with a lot of disks and a 4u additional disk box, you may as well get 4u servers with a lot of disks each, use one for tweets and the other for images. Anyway, images are fairly easy to scale horizontally, there's not much simplicity gained by having them all in one host, like there is for tweets.
Ethernet is fast, you might be able to get in range of DRAM access with an RDMA setup. cache coherency would require some kind of crazy locking, but maybe you could do it with FPGAs attached to the RDMA controllers that implement something like Raft?
it'd be kind of pointless and crash the second any machine in the cluster dies, but kind of a cool idea.
it'd be fun to see what Task Manager would make of it if you could get it to last long enough to boot Windows.
My joke fantasy startup is a cloud provider called one.computer where you just have a slider for the number of cores on your single instance, and it gives you a standard linux system that appears to have 10k cores. Most multithreaded software would absolutely trash the cache-coherency protocols and have poor performance, but it might be useful to easily turn embarrassingly parallel threaded map-reduces into multi-machine ones.
It's relatively easy to have it work slowly (reducing clocks to have a period higher than max latency), but becomes very hard to do at higher freqs.
Beowulf clusters can get you there to some extent, although you can always do better with specialized hardware and software (by then you're building a supercomputer...)
in a cluster, communication isn't real-time. packets drop, fetches fail, clocks skew, machines reboot.
IPC is a gray area. the remote process might die, its threads might be preempted, etc. RTOSes make IPC work more like a single machine, while regular OSes make IPC more like a network call.
so to me, the datacenter-as-mainframe idea falls apart because you need massive amounts of software infrastructure to treat a cluster like a mainframe. you have to use Paxos or Raft for serializing operations, you have to shard data and handle failures, etc. etc.
but it's definitely getting closer, thanks to lots of distributed systems engineering.
I think every big internet service uses user-space networking where required, so that part isn't new.
- spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution. Similar to ML timelines.
- ad relevance: Not a core feature if your costs are low enough. But see the ML estimates for how much throughput A100s have at dot producting ML embeddings.
- web previews: I'd do this by making it the client's responsibility. You'd lose trustworthiness though so users with hacked clients could make troll web previews, they can already do that for a site they control, but not a general site.
- blocks/mutes: Not a concern for the main timeline other than when using ML, when looking at replies will need to fetch blocks/mutes and filter. Whether this costs too much depends on how frequently people look at replies.
I'm fully aware that real Twitter has bajillions of features that I don't investigate, and you couldn't fit all of them on one machine. Many of them make up such a small fraction of load that you could still fit them. Others do indeed pose challenges, but ones similar to features I'd already discussed.
Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering (https://www.jwz.org/blog/2022/11/mastodon-stampede/)
For blocks/mutes, you have to account for retweets and quotes, it's just not a fun problem.
Shipping the product is much more difficult that what's in your post. It's not realistic at all, but it is fun to think about.
"Our approach to blocking links" https://help.twitter.com/en/safety-and-security/phishing-spa...
"The Infrastructure Behind Twitter: Scale" https://blog.twitter.com/engineering/en_us/topics/infrastruc...
"Mux" https://twitter.github.io/finagle/guide/Protocols.html#mux
I do agree that some of this could be done better a decade later (like, using Rust for some things instead of Scala), but it was all considered. A single machine is a fun thing to think about, but not close to realistic. CPU time was not usually the concern in designing these systems.
---
"Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding.
The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons slows down.
Presumably each of those IPs is an instance, none of which share any caching infrastructure with each other, and this problem is going to scale with my number of followers (followers' instances).
This system is not a good system.
Update: Blocking the Mastodon user agent is a workaround for the DDoS. "(Mastodon|http\.rb)/". The side effect is that people on Mastodon who see links to my posts no longer get link previews, just the URL.
---
I personally find this absolutely hilarious. Is that blog hosted on a Raspberry Pi or something? "Over a thousand" requests per second shouldn't even show up on the utilization graphs on a modern server. The comments suggest that he's hitting the database for every request instead of caching GET responses, but even with such a weird config a normal machine should be able to do over 10k/second without breaking a sweat.
Mastodon is written on Ruby on Rails. That should really answer all your questions about the problem but if you're unfamiliar Ruby is slow compared to any compiled language, Rails is slow compared to near-every framework on the planet and it isn't written that well either.
You would be surprised. First off, SSDs are denser than hard drives now if you're willing to spend $$$.
Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-over-Ethernet, etc...
It's fairly easy to get several petabytes of fast storage directly managed by one box. The only limit is the total usable PCIe bandwidth of the CPUs, which for a current-gen EPYC 9004 series processors in a dual-socket configuration is something crazy like 512 GB/s. This vastly exceeds typical NIC speeds. You'd have to balance available bandwidth between multiple 400 Gbps NICs and disks to be able to saturate the system.
People really overestimate the data volume put out by a service like Twitter while simultaneously underestimating the bandwidth capability of a single server.
[0] https://www.liqid.com/products/liqid-elements/liqid-48-port-...
https://www.supermicro.com/en/pressreleases/supermicro-unlea...
Is an infiniband switch connected to a bunch of machines that expose NVMe targets really that different from a SAS expander connected to a bunch of JBOD enclosures? Only difference is that the former can scale beyond 256 drives per controller and fill an entire data center. You're still doing all the compute on one machine so I think it still counts.
We have Zen4c 128 Core with DDR5 now. We might get a 256 Core Zen6c with PCI-E 6.0 and DDR6 by 2026.
I really like these exercise of trying to shrink the amount of server needed, especially those on Web usage. And the mention of Mainframe. Which dont get enough credit for. I did something similar with Netflix 800Gbps's post. [1] Where they could serve every single user with less than 50 Racks by the end of this decade.
[1] https://news.ycombinator.com/item?id=33451430
It's outright comical. Above we have people thinking somehow amount of TLS connections single server can handle is a problem, in service where there would be hundreds of thousands lines of code to generate the content served over it, all while using numbers from what seems like 10+ years old server hardware
Netapp is at something > 300TB storage per node IIRC, but in any case it would make more sense to use some cloud service. AWS EFS and S3 don't have any (practically reachable) limit in size.
Because both are ridiculously slow to the point where they would be completely unusable for a service such as Twitter whose current latency is based off everything largely being in memory.
And Twitter already evaluated using the cloud for their core services and it was cost-prohibitive compared to on-premise.
[1] Flexible Content-based Publish/Subscribe over Programmable Data Planes https://pure.rug.nl/ws/files/206093441/Flexible_Content_base...
I think that this may make sense for some applications, but I also think that if you can utilize software abstractions to improve developer efficiency, it reduces risk in the long run.
[1] https://en.wikipedia.org/wiki/P4_(programming_language)
[2] https://www.intel.com/content/www/us/en/products/network-io/...
I agree with you specialists are expensive but even a team of software engineers runs into the multi million dollar territory.
Why not spend the same amount AND cut down resource use? Hyperscalers have shifted to custom hardware already.
[1] https://github.com/fiberhood/MorphleLogic/blob/main/README_M...
>It is amusing to consider how much of the world you could serve something like Twitter to from a single beefy server if it really was just shuffling tweet sized buffers to network offload cards. Smart clients instead of web pages could make a very large difference. [1]
Very interesting to see the idea worked out in more detail.
[1] https://twitter.com/id_aa_carmack/status/1350672098029694998
Except that's not what it is doing at all.
It assembles all the Tweets internally, applies an ML model to produce a finalised response to the user.
It’s like me running a web crawler on my phone and saying I can replace Google.
The hard part is in being able to translate a search query into a list of pages.
And that requires a level of sophistication that far exceeds a laptop.
Any machine that can do that will be a similar spec to what you need for serving queries. Not as fast as google does it, but a good amount of them.
It's akin to saying the magic behind OpenGPT is the dataset.
The comparable goal to the article is to be a search engine, not to fight google for best results.
George Hotz, is that you ?
Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw.
I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.
I sometimes wonder how much value ML provides vs a proper sort function for anything but advertising.
Do you think that the most successful web companies in the world with arguably the best people i.e. Amazon, Facebook, Instagram, TikTok, LinkedIn, Pinterest, Youtube, Netflix, Snapchat etc. have no idea what they are doing. That the highly complex, expensive and latency impacting recommendation systems could be replaced by trivial sorting.
Or maybe they do work, do translate to increased usage and do significantly impact revenue.
Would it be better to live in a world where Twitter (for example) existed because it is a useful thing and not because it might make lots of money?
It's not just ads, it means the set of people you follow becomes extremely critical for your experience in a way that makes it far less engaging.
That may be good or bad for you as a user depending on what you want, but for Twitter having most people stick to the ML augmented timeline is essential to keep you hooked.
I strongly doubt that entire datacenters need to be used if and only if Twitter obsessively optimized for hardware usage efficiency over everything else. In reality they don't and make some pretty big compromises to actually get stuff built. Hardware is cheap, people are not.
b) No one has said Twitter operates entire data centres.
c) You need more than just NICs and ML accelerators to built a Twitter timeline. You need to rank the content, determine appropriate ads and combine them together. You can't do that in your network card.
They do though. 3 in the US alone (https://www.datacenterdynamics.com/en/news/report-elon-musk-...).
Twitter operates a handful of datacenters because their scale is such that it makes sense.
I find it much more appealing to just make the whole thing run on one fast machine. When you suggest this tend to people say "but scaling!", without understanding how much capacity there is in vertical.
The thing most appealing about single server configs is the simplicity. The more simple a system easy, likely the more reliable and easy to understand.
The software thing most people are building these days can easily run lock stock and barrel on one machine.
I wrote a prototype for an in-memory message queue in Rust and ran it on the fastest EC2 instance I could and it was able to process nearly 8 million messages a second.
You could be forgiven for believing the only way to write software is is a giant kablooie of containers, microservices, cloud functions and kubernetes, because that's what the cloud vendors want you to do, and it's also because it seems to be the primary approach discussed. Every layer of such stuff add complexity, development, devops, maintenance, support, deployment, testing and (un)reliability. Single server systems can be dramatically mnore simple because you can trim is as close as possible down to just the code and the storage.
But it's a pretty objective notation that manually scaled single machines don't scale as well as automation.
If you're just doing a simple application, Sentry really is the way to go, while Datadog and ELK are agent-based and more intended for complex setups and big enterprises (especially in their pricing structure/infra costs).
I'll say that this is a good point, especially because if you don't use containers or a similar solution (even things like shipping VM images, for all I care), you'll end up with environment drift, unless your application is a statically compiled executable with no system dependencies, like a JDK/.NET/Python/Ruby runtime or worse yet, an application server like Tomcat, all of which can have different versions. Worse yet, if you need to install packages on the system, for which you haven't pinned specific versions (e.g. needing something that's installed through apt/yum, rather than package.json or Gemfile, or requirements.txt and so on).
That said, even when you don't use containers, you can still benefit from some pretty nice suggestions that will help make the software you develop easier to manage and run: https://12factor.net/
I'd also suggest that you have a single mechanism for managing everything that you need to run, so if it's not containers and an orchestrator of some sort, at least write systemd services or an equivalent for every process or group of processes that should be running.
Disclaimer: I still think that containers are a good idea, just because of how much of a dumpsterfire managing different OSes, their packages, language runtimes, application dependencies, application executables, port mappings, application resource limits, configuration, logging and other aspects is. Kubernetes, perhaps a bit less so, although when it works, it gets the job done... passably. Then again, Docker Swarm to me felt better for smaller deployments (a better fit for what you want to do vs the resources you have), whereas Nomad was also pretty nice, even if HCL sadly doesn't use the Docker Compose specification.
So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.
And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.
After I've got some experience with Kubernetes, I'd reach for it always because it's very simple and easy to use. But that requires to go through some learning curve, for sure.
The best and unbeatable thing about containers is that there're plenty of ready ones. I have no idea how would I install postgres without apt. I guess I could download binaries (where?), put them somewhere, read docs, craft config file with data dir pointing to anotherwere and so on. That should be doable but that's time. I can docker run it in seconds and that's saved time. Another example is ingress-nginx + cert-manager. It would take hours if not days from me to craft set of scripts and configs to replicate thing which is available almost out of the box in k8s, well tested and just works.
I've seen something similar in projects previously, it never worked all that well.
While the idea of shipping one archive with everything is pretty good, people don't want to include the full JDK and Tomcat installs with each software delivery, unlike with containers, where you get some benefit out of layer re-use when they haven't changed, while having the confidence that what you tested is what you'll ship. Shipping 100 app versions with the same JDK + Tomcat version will mean reused layers instead of 100 copies in the archives. And if you don't ship everything together, but merely suggest that release X should run on JDK version Y, the possibility of someone not following those instructions at least once approaches 100% with every next release.
Furthermore, Tomcat typically will need custom configuration for the app server, as well as configuration for the actual apps. This means that you'd need to store the configuration in a bunch of separate files and then apply (copy) it on top of the newly delivered version. But you can't really do that directly, so you'd need to use something like Meld to compare whether the newly shipped default configuration doesn't include something that your old custom configuration doesn't (e.g. something new in web.xml or server.xml). The same applies to something like cacerts within your JDK install, if you haven't bothered to set up custom files separately.
Worse yet, if people aren't really disciplined about all of this, you'll end up with configuration drift over time - where your dev environment will have configuration A, your test environment will have configuration B (which will sort of be like A), and staging or prod will have something else. You'll be able to ignore some of those differences until everything will go horribly wrong one day, or maybe you'll get degraded performance but without a clear reason for it.
> So IMO it's perfectly possible to run Java applications without containers. You would need to think about network ports, about resource limits, but those are not hard things.
This is only viable/easy/not brittle when you have self-contained .jar files, which admittedly are pretty nice! Though if shipping JDK with each delivery isn't in the cards (for example, because of the space considerations), that's not safe either - I've seen performance degrade 10x because of a JDK patch release was different between two environments, all because of JDK being managed through the system packages.
Resource limits are generally doable, though Xms and Xmx lie to you, you'd need systemd slices or an equivalent for hard resource limits, which I haven't seen anyone seriously bother with, although they're at a risk of the entire server/VM becoming unresponsive should their process go rogue for whatever reason (e.g. CPU at 100%, which is arguably worse than OOM because of bad memory limits).
Ports are okay when you are actually in control of the software and nothing is hardcoded. Then again, another aspect is being able to run multiple versions of software at the same time (e.g. different MySQL/MariaDB releases for different services/projects on the same node), which most nix distributions are pretty bad at.
> And tomcat even provides zero-downtime upgrades, although it's not that easy to set up, but when it works, it does work.
I've seen this attempted, but it never worked properly - the codebases might not have been good, but those redeployments and integrating with Tomcat always lead to either memory leaks or odd cases of the app server breaking. That's why personally I actually enjoy the a...
Containers are a good common denominator because you essentially start with the OS, and then there's a file that automates installing further dependencies and building the artifact, which typically includes the important parts of the runtime environment.
Infrastructure as Code does the same thing for the underlying infra layers and kuberenetes is one of the nicer / quicker implementations of this, but requires you have kubernetes available.Together they largely solve the "works on my PC" problem.
If anything, Kubernetes allows you to save cost by going with a scalable number of small, inexpensive, fully utilized machines, vs one large, expensive, underused one.
It's not. Utilization is a key metric in capacity planning of large scalable apps.
Capacity is based upon max utilization. A scaled web app is does not have constant utilization. The parent I was responding to suggested running on one large/face instance. Ok... if you're capacity planning, are you planning for peak rps or min rps? Obviously peak. Peak times are always a fraction of your total server uptime. This means one big/fast server would be underutilized most of the time.
How do you expect to dynamically vertically scale in cloud to fit demand while using a single server? Re-provision another server (either smaller or larger), redeploy all apps to the server, and then route traffic? Great, you're doing kubernetes job by hand.
Indeed.
Lots of examples out there, one being Let's Encrypt[1] who run off one MySQL server (with a few read replicas but only one write).
[1] https://letsencrypt.org/2021/01/21/next-gen-database-servers...
It is an example. It shows you how you can run a service that issues a few hundred million SSL certs a year off relatively few pieces of hardware, i.e. no need to go drinking the cloud Kool aid.
There will never be a "perfect" example. The overall point here is demonstrating that the first answer to everything doesn't have to include the word "cloud".
> The database alone has multiple machines.
As I said, and the blog says ... there is only one writer. The other nodes are smaller read replicas.
Which again shows you don't need to go with the cloud buzzword-filled database services.
It can't just tolerate being down or having under-load issues like HN often is.
And what happens if that writer goes down. Then the service just stops.
> buzzword-filled
I love how your buzzwords e.g. read replicas are okay but everyone else's are bad.
"100 million certs a year" is only like ~3 certs a second. Maybe twice in peak. That's not much. And you're doing same error, focusing on one core feature and ignoring everything around it
I think you mean vertical, right?
What if your unique machine crash?
Having a system in place that handles most of this gracefully (like kubernetes) is one way of having such a plan, there are others. Which one works best is dependent on your app, cost of downtime, your team that's tasked with bringing everything back up in the middle if the night, etc.
People who leave details like this out when they say "kubernetes is complicated" just haven't seen the complexities of operating a service well.
> the complexities of operating a service well.
Keep in mind that in a lot of business applications, downtime isn't the end of the world and might be an accepted and priced-in "cost" of doing business. Operating it as you consider "well" would just cost them more with no benefit.
It's very simple to make a PoC on a very powerful machine, make it ready from production serving hunderd of millions of users is completely different.
Sounds like you already have quite a number of different containers already.
Several ways of doing this without relying on k8s
> observability
This doesn't require k8s neither and it's more on your app. Systemd can restart systems by itself
> how do you update that stack without bringing down everything
That's probably where redundancy helps the most. I wouldn't run a big service without it (but again it found be at server level)
Can you educate us on how to have a resilient app with no-downtime updates at the server level.
Because if you're doing this via software then it's no different to Kubernetes.
Yeah, k8s might be the easiest way of doing this today if you want only 1 physical server
Nobody should be looking at this and thinking that it’s realistic to actually serve a functional website at this scale on a single machine with actual real world requirements.
Like kafka.
My impression is that it is the serialisation that comes with each service-to-service communication that is really expensive.
https://k3s.io/ makes it really easy to set up, too.
The only thing bad about single server kubernetes is that it'll eat like 1-2 GB of RAM by itself. When you whole server could be 256 MB, that's a lot of wasted RAM.
We have python service which consumes gigabytes of RAM for quite simple task. I'm sure that I'd rewrite it with Rust to consume tens of megabytes of RAM at most. Probably even less.
But I don't have time for that, there are more important things to consider and gigabytes is not that bad. Especially when you have some hardware elasticity with cloud resources.
I think that if you can develop world-scale twitter which could run on a single computer, that's a great skill. But it's a rare skill. It's safer to develop world-scale twitter which will run on Kubernetes and will not require rare developer skills.
As soon as you start accounting for redundancy you have to fan out anyway.
In my experience this ended up with more complicated.
Those systems are typically developed by people who already left and are undocumented, and they become extremely difficult to figure out the config (packages, etc files... oh, where even the service files are located?) and almost impossible to reproduce.
It might be okay to leave it there, but when we need to modify or troubleshoot the system a nightmare begins...
Maybe I was just unlucky, but at least k8s configs are more organized and simpler than dealing with a whole custom configured Linux system.
However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or other in place of the text of the quoted tweet itself but regardless a tweet takes up more than 280 unicode characters.
Also, nowhere in the article are hashtags mentioned. For a system like this to work you need some indexing of hashtags so you aren't doing a full scan of the entire tweet text of every tweet anytime someone decides to search for #YOLO. The system as proposed is missing a highly critical feature of the platform it purports to emulate. I have no insider knowledge but I suspect that index is maybe the second largest thing on disk on the entire platform, apart from the tweets themselves.
Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashtag-only search might fit, mainly because my impression is people doing hashtag searches are a small fraction of traffic nowadays so the main cost is disk, not sure though.
I did run the post by 5 ex-Twitter engineers and none of them said any of my estimates were super wrong, mainly just brought up additional features and things I didn't discuss (which I edited into the post before publishing). Still possible that they just didn't divulge or didn't know some number they knew that I estimated very wrong.
>possible that they just didn't divulge
0. https://news.sky.com/story/amp/jersey-tower-explosion-questi...
Absence of evidence, in your case via a clean building inspection, does not mean the building is safe. It just means the checklist of known items was considered and nothing bad found.
Ask a building inspector if their clean report proves nothing is wrong with the building.
They will be firm and quick to inform you that it’s not a warranty — anything not checked was not covered. Items not covered could still be significant problems.
That’s the whole point of the saying. Absence of evidence is not evidence of absence.
Another poster dug into some implementation details that I'm not going to go into. I think you could shoehorn it into an extremely large server alongside the rest of your project but then you're looking at processing overhead and capacity management around the indexes themselves starting to become a more substantial part of processing power. Consider that for each tweet you need to break out what hashtags are in it, create records, update indexes, and many times there's several hashtags in a given tweet.
When I last ran analytics on the firehose data (ca. 2015/16) I saw something like 20% of all tweets had 3 or more hashtags. I only remember this fact because I built a demo around doing that kind of analytics. That may have changed over time obviously, however without that kind of information we don't have a good guesstimate even of what storage and index management there looks like. I'd be curious if the former Twitter engineers you polled worked on the data storage side of things. Coming at it from the other end of things, I've met more than a few application engineers who genuinely have no clue how much work a DBA (or equivalent) does to get things stored and indexed well and responsively.
Also, the big data storage isn't text, it's images and videos.
While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.
It's often fine to start without a fully decoupled system (net present value of the time and money needed to scale out might be far too high), but you need to know whether or not it's likely to come and what to look for so you can start preparing in time.
What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database.
So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously.
I also heard that a method for spamming Twitter trending topics is to send tweets and delete them quickly.
My impression is that Twitter is big on real time processing. They definitely don't search the entire database for #YOLO tweets, instead they seem to be searching the almost-live stuff and some archived stuff(probably ranked and saved as noteworthy tweets or something).
I really wonder how much of a challenge this is and how much it occupies, not even talking about disk, but continuing the theoretical exercise in the linked URL, you can get 1U size servers with 2TB of RAM these days.
Adding images, videos, other large attachments, rich search, and all the advertising and billing and analytics stuff would blow this out of the water, but... maybe not by as much as people think...? I would not be surprised if a very performance-engineered version of Twitter could run on a few dozen racks full of beefy machines like this with HPC-grade super-fast fabric interconnects.
I have a strong sense that most large scale systems are way less efficient than what's possible. They trade development ease, modularity, and velocity for performance by using a lot of microservices, flabby but easy and flexible protocols (e.g. JSON over HTTP), slow dynamic languages, and layers of abstraction that could be integrated a lot more tightly.
Of course that may be a rational trade-off if velocity, flexibility, and labor costs and logistics matter more than hardware, power, or data center floor space costs.
While some of those writes may well be acceptable to lose, letting you write to caches, effectively you need to assume there are more analytics events triggering writes to something than there are tweet views.
Everyone has to be employed so it's better to keep adding more crap to products and make those products disposable in order to give people a job.
Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)
You'd definitely need at least two servers. But I think you could surely just have simple master/slave replication and switch between them.
Personally I wouldn't run a critical service on only one server, but two servers? Definitely doable. I actually have a service running on two servers in different DCs 700 miles apart. Zero downtime in 9 years. :)
With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes.
Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.
Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day?
> Realistically, there would be a mirroring server to be prepared for hardware failures. One server can be restarted while the other is the main server.
But for that mirroring, you need to replicate disk writes, databases, backups etc.. This additional load would easily bring the server to a point where a single server would no longer suffice, even an insanely spec'ed one.
We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.
Who needs Twitter to be a service without any downtime?
Source: https://foreignpolicy.com/2009/06/16/state-department-interv...
You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude.
EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be cheaper than buying a second server in any reasonable timeframe.
(You are right, servers are cheap compared to employee costs though.)
Yes, this would be big enough to need specifically factoring into a real implementation design. But it would not be big enough to invalidate the proposed idea so I understand leaving it off, at least initially, to simplify the thought experiment.
Similarly to support a message responding to, or quoting, a single other you only need one or two NULLable ID references, 16 bytes per message, which will likely be dwarfed by the message text in the average case. Given it likely makes sense to use something like SQL Server's compression options for data like this the average load imposed will be much smaller than 16 bytes/message.
We are fiddling, fairly insignificantly, measurable but to massively, with constants a & b in O(a+bN) here, so the storage problem is still essentially of order O(N) [where N is total length of the stored messages].
I'd probably go as far to say that the indexes _generally_ at twitter could be larger than the tweets
The data structures for the @BeefWellington timeline of tweets and the one for the #BeefWellington timeline of tweets could look roughly the same.
I want to add another concept that may impact, considerably, the storage, which is "threads". I'm not sure what is the percentage of threads/tweets but what I consider an important factor is that threads do not have a maximum number of characters.
Text search, hashtag index, some structured data for popular tweets, etc...
In order to deliver search results I would not be surprised if tweets are duplicated/denormalized, for quick search/lookup.
When interviewing developers I always ask them what is the largest public web site they ever worked on and then probe about performance issues they encountered and how they resolved them in order gague how far along they are in their skill development.
I would never plan to run a production service on a single server just because coordinating changes in the active dataset among two or more production servers often changes your design significantly, and you want to plan for that because the consumer grade hardware we all use has a nasty habit of not working after power cycles (which still tends to be the most strain a system goes through, even in a world of SSD storage).
There are storage size issues (like how big is their long tail; quite large I'd imagine), but its a fun thing to think about.
[1] https://people.freebsd.org/~gallatin/talks/euro2022.pdf
You’d also have to quantify the improved user experience from seeing less spam v.s. inflated ad revenue for garbage views / content.
HTTP with connection: keep-open can serve 100k req/sec. But that's for one client being served repeatedly over 1 connection. And this is the inflated number that's published in webserver benchmark tests.
For more practical down to earth test, you need to measure performance w/o keep-alive. Request per second will drop to 12k / sec then.
And that's for HTTP without encryption or ssl handshake. Use HTTPS and watch it fall down to only 400 req / sec under load test [ without connection: keep-alive ].
That's what I observer.
I'm talking about a hypothetical HTTPS server that used optimized kernel-bypass networking. Here's a kernel-bypass HTTP server benchmarked doing 50k new connections per core second while re-using nginx code: https://github.com/F-Stack/f-stack. But I don't know of anyone who's done something similar with HTTPS support.
"Quant uses the warpcore zero-copy userspace UDP/IP stack, which in addition to running on on top of the standard Socket API has support for the netmap fast packet I/O framework, as well as the Particle and RIOT IoT stacks. Quant hence supports traditional POSIX platforms (Linux, MacOS, FreeBSD, etc.) as well as embedded systems."
One thing which we noticed was that there was a considerable difference in performance characteristics based on how we parallelized the load testing tool (multiple threads, multiple processes, multiple kubernetes pods, pods forced to be distributed across nodes).
I think that when you run non-distrubuted load tests you benefit from bunch of cool things which happen with http2 and Linux (multiplexing, resource sharing etc) which might make applications seem much faster than they would be in the real world.
I'm running about 2000 requests/s in one of my real-world production systems. All of the requests are without keep-alive and use TLS. They use about one core for TLS and HTTP processing.
I have a basic LAMP server running on a 4-core VM on a laptop. I just threw ApacheBench at it (not the fastest benchmarking tool, either -- it eats up 1 core all by itself), and it handles 1200 req/s TLS with no keepalive, and 3400 req/s with keepalive. This stuff scales linearly with core count, so I wouldn't be surprised to see much higher numbers in real servers.
Here's one I ran into recently: if a range has only 1 of 3 replicas online then it will stop accepting traffic for that range until it has 3 replicas again.
(for the folks at home, "range" is a technical term for 512 bit slice of the data - CRDB replicates at the range level)
So, in some code I wrote, I had account for not only 1) the whole DB being unavailable but also 2) just one replica being unavailable (they're different failure modes that say different things about the health of the system).
It's a good behavior! Good for durability. But I had to do some work to deal with it, spend an hour coming up with a solution, etc. There are databases that work at Twitter scale but no there are no silver bullets among those that do. You need full time engineers to manage the complexity and keep it online, or else it could cost the company shitloads of money - I've seen websites of similar scale where a two-hour outage cost them $20 million.
Like last time, and the time before that, and the time before that, and the time before that.
I understand the frustration with flavor of the week "best practices" and the constant churn of frameworks and ideas, but software engineering as a practice IS moving forward. The difficulty is separating the good ideas (CI/CD, for example) from the trends (TDD all the things all the time) ahead of time.
In fact it’s often meant to distract from the lack of progress.
It’s not the lack of progress that is the concern, it’s the subterfuge.
I see a lot of comments here assuming that this proves something about Twitter being inefficient. Before you jump to conclusions, take a look at the author’s code: https://github.com/trishume/twitterperf
Notably absent are things like serving HTTP, not to even mention HTTPS. This was a fun exercise in algorithms, I/O, and benchmarking. It wasn’t actually imitating anything that resembles actual Twitter or even a usable website.
I'm now somewhat confident I could implement this if I tried, but it would take many years, the prototype and math is to check whether there's anything that would stop me if I tried and be a fun blog post about what systems are capable of.
I've worked on a team building a system to handle millions of messages per second per machine, and spending weeks doing math and building performance prototypes like this is exactly what we did before we built it for real.
Of course. I was commenting here to counter all of the comments declaring that this proves Twitter doesn’t need all of their servers, etc.
It’s a fun article, but the comments here interpreting it as proving something about Twitter engineers being bad are kind of depressing.
However, the stateless workload can still operate in a read-only manner if the stateful component failed.
I run an email forwarding service[1], and one of challenge is how can I ensure the email forwarding still work even if my primary database failed.
And I come up with a design that the app boot up, and load entire routing data from my postgres into its memory data structure, and persisted to local storage. So if postgres datbase failed, as long as I have an instance of those app(which I can run as many as I can), the system continue to work for existing customer.
The app use listen/notify to load new data from postgres into its memory.
Not exactly the same concept as the artcile, but the idea is that we try to design the system in a way where it can operate fully on a single machine. Another cool thing is that it easiser to test this, instead of loading data from Postgres, it can load from config files, so essentially the core biz logic is isolated into a single machine.
---
https://mailwip.com
https://gist.github.com/jboner/2841832
Essentially IO is expensive except within a datacenter but even in a data center, you can do a lot of loop iterations in a hot loop in the time it takes to ask a server for something.
There is a whitepaper which talks about the raw throughput and performance of single core systems outperforming scalable systems. These should be required reading of those developing distributed systems.
http://www.frankmcsherry.org/assets/COST.pdf A summary: http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/
How much RAM did your advertising network need? Becuase that is what makes twitter a business! How are you building your advertiser profiles? Where are you accounting for fast roll out of a Snapchat/Instagram/BeReal/Tiktok equivalent? Oh look, your 140 characters just turned into a few hundreds megs of video that you're going to transcode 16 different ways for Qos. Ruh Roh!
How are your 1,000 engineers going to push their code to production on one machine?
Almost always the answer to "do more work" or "buy more machines" is "buy more machines".
All I'm saying is I'd change it to "Toy twitter on one machine" not Production.
> How are your 1,000 engineers going to push their code to production on one machine?
That might actually be the reason why Twitter barely keeps afloat. 1k engineers for a product that's already built and hasn't fundamentally changed nor evolved in years makes me wonder what business value those engineers actually provide.
That was some time ago, though.
9 web servers to serve the entire network. I wish more developers were aware of just how performant modern (non-cloud) hardware is.