Often people will say “AWS is so much bigger than us, they have to accommodate us / what could we know that they don’t” is that AWS is still building for the median customer (+/-). It doesn’t make financial sense to optimize for the extremes and extremes can be extreme (per the tweet thread). If you’re a reasonably technical company there’s a good chance something about your main workloads will challenge AWS.
More than laziness, it's too complex a problem for most decision makers. Environment is ever changing and extremely competitive. Most people don't handle such problems well and few get the training or have the support. It's pretty much like crypto. Most people are sold shit they have no capacity to understand.
So the story writes itself.
These are info asymmetry problems and society and business hasn't been dealing with them at such scales for too long. So solutions are a work in progress.
The cloud is great when you are starting out, but at scale you are going to hemorrhage money. This is why it's dangerous to tie yourself to all those cloud "toys" and then rapidly expand.
If I just want to have my own little home page, a micro EC2 instance will do just fine - I don't want to go to a colocation for that.
Once you are at scale, you need to get out of the cloud. I agree with DHH on this.
m6.12xl are 48 vCPU systems. 25 million cores is a hilariously huge amount of compute to use to run a site of twitter size, which I guess does point to how much tech debt they have ignored over time.
There is nothing simple in data moving. Problem is that it can be abstracted away so you do not have to worry about it. Out of sight out of mind. Example - majority of the CPU area is devoted to data moving problem (cache, branch prediction, out of order execution, register renaming). The same is happening on other levels of the stack (OS, network and so on).
But they do serve videos, which possibly they have to transcode or otherwise process on import (?)
They probably also have some filters for pictures to remove porn and stuff like that.
This can amount to serious CPU if you have millions of users uploading that stuff daily.
Developing for the cloud makes sense when you're developing features as fast as you can leveraging cloud services to make it all easier. When you have a relatively stable feature set and scaling massively it's better to use private datacenters.
Snapchat pays Google cloud $400MM a year. Using these cost metrics, are Twitter's compute needs 9X those of Snapchat? maybe so. seems reasonable to me. And then perhaps, this person is saying that cloud (Google or otherwise) is not the way to go for Snapchat
300m/month to deliver text, some photos and a bit of video to (admittedly tons of) people sounds insane.
Then again based on the architecture sketch Elon posted https://mobile.twitter.com/elonmusk/status/15938990295318036... most of the compute is being used by ... the feed recommender AI crap that works awfully anyway? Insanely bad job programming this product.
29 comments
[ 0.24 ms ] story [ 60.9 ms ] threadThere’s tons of much, much more powerful and cheap servers you can get online than those of the major clouds, often with little or no traffic fees.
Most companies are too lazy to even investigate the choices.
These are info asymmetry problems and society and business hasn't been dealing with them at such scales for too long. So solutions are a work in progress.
If I just want to have my own little home page, a micro EC2 instance will do just fine - I don't want to go to a colocation for that.
Once you are at scale, you need to get out of the cloud. I agree with DHH on this.
https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...
Twitter is the 4th most visited site in the world.
https://en.wikipedia.org/wiki/List_of_most_visited_websites
But they do serve videos, which possibly they have to transcode or otherwise process on import (?) They probably also have some filters for pictures to remove porn and stuff like that.
This can amount to serious CPU if you have millions of users uploading that stuff daily.
A pure text-based Twitter would be much smaller.
https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spen...
Then again based on the architecture sketch Elon posted https://mobile.twitter.com/elonmusk/status/15938990295318036... most of the compute is being used by ... the feed recommender AI crap that works awfully anyway? Insanely bad job programming this product.
And I think HN still runs on one server? 500,000 for twitter seems a tad inefficient