Ask HN: How long could Twitter operate if the remaining staff quit?
In almost every organization there are roles that could be trimmed, but it seems extreme that Twitter is potentially laying off half the company today.
Can the company survive that? If the remaining team is over worked or feels like they are not respected… what would happen if they all left en mass?
53 comments
[ 2.3 ms ] story [ 117 ms ] threadI greatly over-simplified the answer but this topic is worthy of a book.
Is it tho? In any company small and large once ownership of high level management changes hiccups can happen.
Only if one wanted a more thorough and in depth picture of what this typically looks like day-to-day, week-to-week. Hiccups happen but when a company has been around for a long time and processes are very entrenched, a take-over can disrupt business processes and work flows enough for customers/consumers to feel it. The business will learn with time what undocumented relationships it has/had with vendors and what contractual obligations must be met that may not have been accounted for. Someone will discover for example some message-queue server that nobody documented or knew about when it finally fails and turns out all of Twitter depends on it. This is not hypothetical, I found two very old workstation-class "servers" under a CTO's desk in a financial company
[Edit] I should also add that since this is a take-over and not a chapter 11 reorg, there are no legal protections for contractual obligations so all of those will have to be discovered and addressed. Not doing so in a timely manor can be extra bumpy or costly. One can only hope they had good record keeping.
Experienced leaders that have been through mergers and take-overs should not really face any problems they can not solve, but lesser experienced leaders will have opportunities to make some monumental and profound mistakes. This is the gap that leaders could benefit from reading books on the topic and even then some leaders will misinterpret the intentions of the books or will not get the lessons to be learned but it is still better than trying to apply the logic from ones past experiences alone. Leadership techniques and business logic that worked at one company may not translate appropriately to Twitter. Along that line I should also add that leaders should not literally use such books as step-by-step instruction manuals but rather they should apply some critical thinking how does this translate to the current situation.
When they need someone to fill a specific role, they will no doubt go to the nearest consultancy and spend 5x as much on temporarily filling the role.
Or... rehire the same roles on a lower salary.
The problem happens when something breaks - let's say your database changes the execution plan and it crawls to a halt. I've seen this happen out of the blue and it was a few hours of work to figure out a solution. If you don't have a developer who understands the code, you might be sunk.
Something that autoupgrades would become out of spec, refuse to start due to a change in config files.
A counter would roll over somewhere.
A disk fills up.
A cert expires. A domain name entry expires.
Could be days/weeks/months.
Of course they'd actually be pulled down by law enforcement much quicker than that if all their content mods left.
It's messy but there are pretty straightforward levers to pull.
Most teams need about 10% of staff for skeleton maintenance. You can last a little bit with one staff to restart things as they go, but you'll have a quick degradation of service. If certain microservices fail, it may be the advertizing not working right or something failing with billing or moderation or some other internal tool, but a user might just notice that something is slightly wonky.
However, you won't be able to change much at that point. Changes become exceedingly difficult because of the maintenance overhead.
I think the real issue is that without heavy content moderation it'll quickly get overrun with spam. A great example is to go find an abandoned subreddit and see what bad shape it's in.
With some of their service issues in the past few weeks (DC stuff that's been publicized), it could be less. To the very best of my knowledge no substantial resiliency work has been done recently.
I do know that a good number of folks on-call for core services were laid off while on-call, so, that bodes well. I feel bad for everyone left trying to keep things running.
One is that hardware fails and needs to be replaced. That requires people who know how to install the replacement hardware and deploy to it. That's assuming the new hardware is 100% compatible - that won't be the case for more than a handful of years.
Another is currently unknown security vulnerabilities, whether in their own code or in external packages they use. Those vulnerabilities are there and they will be discovered. Once they are, things start being taken down from the outside until the system collapses.
Yet another is bugs. Every system of this scale has a large number of bugs, many of them unknown. Some of those won't be discovered until the right conditions arise - the right combination of data, timing, etc. When they are finally triggered, some of those bugs will take down entire subsystems, some of which are critical to the product functioning.
There are many more examples like this. There is no such thing as indefinite resiliency for anything near this scale.
The modern version of this (kubernetes + AWS/GCP), if designed could likely continue to run for a long long time. Especially a product as simple as twitter.
Unlike your Solaris box, they are the target of constant advanced hacking attempts. I've been a part of the response when AWS was doing urgent work because of a security incident. The company I worked at was large enough to be paying AWS over a $1M a month when one such incident required dozens of our engineers working around the clock for three days to deal with AWS's response. We weren't even directly involved in the security issue. But without that engineering effort, our product would have shut down. There were other security incidents we were directly involved in and those would have taken us down without an even bigger response (whether or not we were running in AWS).
And then there are hardware failure rates. Hard drives alone fail at a rate of 1-2% per year[0]. Not a big deal on a single box. A very big deal when you have many thousands of hard drives - multiple drives fail every day. Unless you want to WAY over-allocate storage for redundancy. Even with that, there are surprising vulnerabilities to hardware failure at this scale.
----
[0]https://www.backblaze.com/b2/hard-drive-test-data.html
* Live migrate won't upgrade the CPU family you're running on, so eventually someone/a something on your end will be forced to deal with migrating it, but that's O(years).
If development stops and the model is fixed bad actors can find the gaps given enough time
Now consider sales, content moderation, HR, lawyers.
Now multiply that by multiple countries that need servicing.
7500 people is 750 teams of ten. It's really a rather moderate engineering organization for a problem of their scale.
But coming to Twitter engineering teams, what are they doing actually? their primary technical product is tweet timelines and trends, their mobile apps are kind of okay, the user experience hasn't changed much except getting slower and slower. If all they do is rewrites, they are simply wasting the talent.
There could be SRE, Security, Datacenter admins running 24x7 but other than that, I see no useful idea out of them in a long long while. See instagram, snapchat and others for example, they've explored the unknown and brought us some quite good and plenty bad social media. Twitter at least should've been masters in content moderation by now, but they're far from it.
For an outsider they seem to have lost their way and are being a couch potato munching on the junk food.
The way we engineers build systems is to have a solid core and then a lot of fancy cruft to handle the constant stream of inevitable changes that bring in new features or update stuff. Take the changes away by removing people who’d roll them out, and you have the core running as it was designed to do.
I believe when the remaining engineers or the new ones(if they will be..) start rolling out changes is when shit will break and make the whole thing wildly unpredictable.
No system is perfect. It's how you respond to them that matters and if no human is in the loop what sort of response can you expect?
I would be shocked if they actually had a 50% layoff, from the news reports and everything I've seen on social media, it's not that deep otherwise there would be far larger reports of it by now. If it turns out that 50% number is wrong, I think that would really open the eyes of Twitter employees as to how much the media is spreading lies about Musk.
If it turns out the 50% number is true, I'm pretty sure the company won't survive. Having been through layoffs, the rest of the employees of such a huge cut will likely not work or not even care and immediately start looking for a new job because they know they are next. That's not a wise way to run a company, in my opinion.
Also, I don’t think the majority of people even work on core products.. for example, did you know Twitter implemented their own OnlyFans competitor? It never launched afaik
Step 1 is to trim as much as possible and consolidate core functionality…
Step 2 is back to profitability.
I understand that, let’s see if it works
They might slow down but not stop operating
[0] https://s3.documentcloud.org/documents/22186683/twitter-whis... page 28
Yes, unless the folks in charge are incredibly incompetent.
> what would happen if they all left en mass?
For Twitter, that is an "...if a giant asteroid vaporized San Francisco" fantasy scenario. The remaining workers are not a clone army, and vary in all sorts of ways. Including what it would take to convince them to leave ~immediately. Elan has years of experience leading very large, technology-oriented companies - he is quite aware of the wide, grey optimization zones around "be very demanding of and difficult with your employees".
1. Layers of management and organizational dynamics push up the number of people. There is a correlation between the number a manager manages and his status in the company. The larger the team - the larger your budgets are - the more important you are. Internal goals (to rise in the organization) override external goals (sell and generate profits). When owners are leaving the company or have already cashed out, their motivation to keep financial discipline is lowered and management quality decreases. Owners differ from the hired managers because they prioritize external goals (sales, profit, long-term company health) not their own personal employee goals.
2. There is a strange fallacy that when you get your VC capital you need to increase your headcount to be perceived as a "serious" company. If you are 10 people working remotely you won't be able to ask for the same valuation in your second or third round as a "looks like a big company" startup with an office and 100 employees. Investors would push (sometimes inadvertenly) founders to grow personel at higher rate, because investors sell the company to other investors. They package the company to exit, they don't care that much about sales and profitability except as a baseline numbers for the next round valuation. I've seen this several times.
The contrapoint is that Instagram had 13 people when it was sold to Facebook and Whatsapp had around 40 people.
Twitter has 7500 employees. There's got to be a lot of layers of management, processes that can be removed from the company. For Elon, fixing cashflow now is more important that keeping everything working smoothly. It works smoothly to generate loss every quarter. It's easier to cut the losses than generate revenue. It makes perfect sense.
And that's before a massive drop[1] coming from how badly he's handled the acquisition, as well as his personal reputation.
0. https://www.nytimes.com/2022/10/30/technology/elon-musk-twit...
1. https://nypost.com/2022/11/04/elon-musk-says-twitter-has-see...
These things can be renegotiated, it's hard but doable. And we can't be sure how focused Elon would be on Twitter at all. I doubt it and things are not looking good for Twitter long-term.
He could've be forced to buy it and now it's minimizing future prolbems for him personally.