Flagged - title is editorialized. Correct title is "Interview with Raffi Krikorian on Twitter's Infrastructure". Edit: This has been fixed now. Thanks.
This actually seems like a disguised recruitment piece, until the last answer where it plainly is. There are little to no actual technical details about their SOA implementation other than handwaving about splitting their monolithic app into services and the benefits it brings. I guess it's just meant to entice developers with stories of the awesome solutions they will get to maintain as a cog in Twitter.
Spent time reading it only to find it to be a recruitment fluff piece devoid of any useful information at all. Angry at myself for even bothering to finish it...
Let me split the difference between you and the other replies to you: it's classic tech-blog material. The journalist mistakes "id's" with "ideas" several times, and the engineer being interviewed is happy to share the fruits of his labor.
You, on the other hand, seem to care way too much about how others perceive Twitter. Why?
I have always thought twitter TPS were high, until i heard China WeiBo was doing those TPS constantly and had a highest TPS record 10x more then twitter..
Fanout is soft realtime, but yeah, we don't block lady gagas tweets to 40 million people until they're all delivered. Note that she counts as 1 in the tps calculation when she tweets. Btw, we're hiring if you'd like to help us make twitter even better.
Not sure what the original comment was, but essentially trading firms build their own ticker plants to get their "curated" view. Obviously, twitter followers won't be building their own tweet plants. I'd imagine that the main challenges that are faced with twitter's fan out is in data persistence and scale of data involved.
We could talk about how trading system architecture does not apply, but there are potentially many ways that it does. The main difference in web architecture versus trading-system architecture is the messaging data model. Trading systems are architected for low-latency by building messaging systems that focus on transmitting events as opposed to fully fleshed-out data structures. For the uninitiated, a typical web-app will commit data to a DB, then request all data. A low-latency trading system does not build an orderbook data structure and commit that to a DB, then request it from the DB, it merely listens to a stream of events (e.g. firehose) consisting of buys, sells & executions and builds its own book, so that the orderbook data-structure is always available locally in-memory as opposed to being a DB request away. The beauty with this architecture is that special data structures need not be a complex select/join away with DB's and cache layers taking on a ton of work. It allows just the events to be propagated and let each app decide for itself what it necessary and what isn't necessary and it allows the relevant data to be built in realtime, with low-latency and a high amount of determinism.
Modern systems on higher-end commodity hardware that I have dealt with, with the right tuning can push around 2M 120Byte msgs/sec with an average latency of 6-7 microseconds per host. I'm told with extra/better tuning hitting 3M 120Bytes messages/sec sustained per host. Of course this is doing everything in-memory w/out disk, but even having a disk as a bottleneck doesn't preclude this kind of performance. It just introduced 5-20ms outliers in the message processing workflow as pages are flushed to disk.
The way I've envisioned applying trading system arch to Twitter is where there are tweet submitters that get aggregated into a sequencer that then a bunch of fan-out machines read and committing the fanned out data structures to a memcache and/or data persistence cluster of some type. Each fan-out can be its own shard discarding irrelevant tweets. Other apps that want data can get them via other apps that provide a firehose listening directly from the sequenced data from the firehose or merely subscribing/joining the firehose feeds and processing the data appropriately.
This should be able to handle the hundreds of millions of fanouts per second that are necessary in a distributed manner with great efficiency in the number of hosts involved. If you need to expand capacity, it's just a matter of bringing up extra capacity to the fan-out hosts and persister hosts.
18 comments
[ 3.2 ms ] story [ 52.9 ms ] threadThis actually seems like a disguised recruitment piece, until the last answer where it plainly is. There are little to no actual technical details about their SOA implementation other than handwaving about splitting their monolithic app into services and the benefits it brings. I guess it's just meant to entice developers with stories of the awesome solutions they will get to maintain as a cog in Twitter.
You, on the other hand, seem to care way too much about how others perceive Twitter. Why?
I decided this comment wasn't constructive.
We could talk about how trading system architecture does not apply, but there are potentially many ways that it does. The main difference in web architecture versus trading-system architecture is the messaging data model. Trading systems are architected for low-latency by building messaging systems that focus on transmitting events as opposed to fully fleshed-out data structures. For the uninitiated, a typical web-app will commit data to a DB, then request all data. A low-latency trading system does not build an orderbook data structure and commit that to a DB, then request it from the DB, it merely listens to a stream of events (e.g. firehose) consisting of buys, sells & executions and builds its own book, so that the orderbook data-structure is always available locally in-memory as opposed to being a DB request away. The beauty with this architecture is that special data structures need not be a complex select/join away with DB's and cache layers taking on a ton of work. It allows just the events to be propagated and let each app decide for itself what it necessary and what isn't necessary and it allows the relevant data to be built in realtime, with low-latency and a high amount of determinism.
Modern systems on higher-end commodity hardware that I have dealt with, with the right tuning can push around 2M 120Byte msgs/sec with an average latency of 6-7 microseconds per host. I'm told with extra/better tuning hitting 3M 120Bytes messages/sec sustained per host. Of course this is doing everything in-memory w/out disk, but even having a disk as a bottleneck doesn't preclude this kind of performance. It just introduced 5-20ms outliers in the message processing workflow as pages are flushed to disk.
The way I've envisioned applying trading system arch to Twitter is where there are tweet submitters that get aggregated into a sequencer that then a bunch of fan-out machines read and committing the fanned out data structures to a memcache and/or data persistence cluster of some type. Each fan-out can be its own shard discarding irrelevant tweets. Other apps that want data can get them via other apps that provide a firehose listening directly from the sequenced data from the firehose or merely subscribing/joining the firehose feeds and processing the data appropriately.
This should be able to handle the hundreds of millions of fanouts per second that are necessary in a distributed manner with great efficiency in the number of hosts involved. If you need to expand capacity, it's just a matter of bringing up extra capacity to the fan-out hosts and persister hosts.