>So at large company like Twitter, the technology bears these battle-scars, these overgrown jungles, these spots in the code clearly labelled "Here There Be Dragons" where wise developers fear to tread (and where unwise developers are eaten), and the only thing keeping the whole damn thing running despite all of this... are the people. There are landmines and tripwires and countdown bombs all over that tech stack, and Elon Musk permanently sent home 50% of the people who know where any of it is or how to deal with it.
Yeah, whaetever, as devops for Twitter is some huge mystery, or the codebase is so deep.
This isn't Photoshop or Clang, it's a glorified IM app.
Edit: The post is flagged dead, so answering here:
> It's a glorified IM app running at global scale.
Yes. So like every major web app these days, which is literally thousands of them.
> the problems of maintaining a social graph and recommendation engine at that scale would make Twitter significantly more complex than a standard instant messaging app, since all the data can’t be stored in a simple distributed k-v store.
Yes. Still a very specific and narrow problem, with clear cut solutions, also solved by thousands of others companies and their teams (hec, for some time now, infinite scalability is available as a capability to small startup teams, through AWS, GAE, Azure, Cloudflare, and so on).
> Yeah, whaetever, as devops for Twitter is some huge mystery, or the codebase is so deep.
This isn't Photoshop or Clang, it's a glorified IM app.
So in my perspective (and similar but not exactly same experience), the problems of maintaining a social graph and recommendation engine at that scale would make Twitter significantly more complex than a standard instant messaging app, since all the data can’t be stored in a simple distributed k-v store. Even one of the developers at WhatsApp said so. Do you have a different experience?
Once you run at global scale like Twitter does, maintaining even a TODO app introduces horrifying new problems you hadn't considered before.
Couple this with a drive to move quickly, and the solutions implemented frequently aren't, well, _good._ At a minimum, they're not robust, and require specialized knowledge to keep running.
So while the problem being solved isn't terribly technically difficult, from like a 100k foot view, the scale of the solution required plus the decade's worth of hacks and permanent "temporary" solutions ends up being incredibly complex.
EDIT: Not sure why the post is flagged, it doesn't seem to break any guidelines, but sure okay. Maybe too much Twitter news recently.
Anyway, also replying here, since parent edited their comment to reply.
I'm not arguing that Twitter is special or unique or anything, or that their problems cannot be easily solved in a vacuum. The problem is that Twitter is not in a vacuum, they've got a decade of systems that grew in fits and spurts. They don't have common, easy-to-administer stuff across their whole stack, they've got weird, custom stuff in critical paths.
And more businesses are like this than you think. Businesses don't grow cleanly, they don't adopt technology cleanly. They grow with business needs, and they grow usually with less time allotted than you'd want to do things right.
As a consequence, while something _like_ Twitter would be simple to build in concept, the _actual_ systems running at Twitter are going to be very specialized and require special knowledge to run efficiently.
4 comments
[ 3.0 ms ] story [ 25.0 ms ] threadYeah, whaetever, as devops for Twitter is some huge mystery, or the codebase is so deep.
This isn't Photoshop or Clang, it's a glorified IM app.
Edit: The post is flagged dead, so answering here:
> It's a glorified IM app running at global scale.
Yes. So like every major web app these days, which is literally thousands of them.
> the problems of maintaining a social graph and recommendation engine at that scale would make Twitter significantly more complex than a standard instant messaging app, since all the data can’t be stored in a simple distributed k-v store.
Yes. Still a very specific and narrow problem, with clear cut solutions, also solved by thousands of others companies and their teams (hec, for some time now, infinite scalability is available as a capability to small startup teams, through AWS, GAE, Azure, Cloudflare, and so on).
So in my perspective (and similar but not exactly same experience), the problems of maintaining a social graph and recommendation engine at that scale would make Twitter significantly more complex than a standard instant messaging app, since all the data can’t be stored in a simple distributed k-v store. Even one of the developers at WhatsApp said so. Do you have a different experience?
Once you run at global scale like Twitter does, maintaining even a TODO app introduces horrifying new problems you hadn't considered before.
Couple this with a drive to move quickly, and the solutions implemented frequently aren't, well, _good._ At a minimum, they're not robust, and require specialized knowledge to keep running.
So while the problem being solved isn't terribly technically difficult, from like a 100k foot view, the scale of the solution required plus the decade's worth of hacks and permanent "temporary" solutions ends up being incredibly complex.
EDIT: Not sure why the post is flagged, it doesn't seem to break any guidelines, but sure okay. Maybe too much Twitter news recently.
Anyway, also replying here, since parent edited their comment to reply.
I'm not arguing that Twitter is special or unique or anything, or that their problems cannot be easily solved in a vacuum. The problem is that Twitter is not in a vacuum, they've got a decade of systems that grew in fits and spurts. They don't have common, easy-to-administer stuff across their whole stack, they've got weird, custom stuff in critical paths.
And more businesses are like this than you think. Businesses don't grow cleanly, they don't adopt technology cleanly. They grow with business needs, and they grow usually with less time allotted than you'd want to do things right.
As a consequence, while something _like_ Twitter would be simple to build in concept, the _actual_ systems running at Twitter are going to be very specialized and require special knowledge to run efficiently.
Will be curious how this plays out, but based upon some recent news, does not look good at all for twitter