So one theory about Musk’s approach is that you cut down twitter to just it’s core functionality. So what do those other 80% of micro services do? Content moderation, spam, analytics. Presumably some of that infrastructure is related to advertising. But so far all of his numbers seem random and inflated, but I’m guess his strategy is going to be shut a bunch of stuff off and see what you don’t need. Fair enough when they’re in the dire financial straights they are.
Is it just me though or is part of this bike shedding on micro services vs monolith. Not really enough information to determine.
Edit: I hope all their circuit breakers work correctly.
Yup. Thing is though, nothing about their financial situation justifies this level of recklessness. Yes, there’s an urgent need to cut costs and find new sources of revenue, thanks to Musk’s own decisions to perform a leveraged buyout and then scare off advertisers. But that doesn’t justify making such sweeping changes in so short a time that there’s no way a proper due diligence is being done.
Honestly, I suspect the issue is much simpler. I think Musk detests and can’t bring himself to respect Twitter’s engineers at all, thanks to preconceptions he developed on the outside. That’s why he’s so happy to mock and trash their work in public, why he wanted to get rid of half of them without even giving them the courtesy of speaking to them, and why he believes he knows better. Expect most of those remaining to be replaced as soon as Musk finds other willing saps to take their places.
It’s the same reason so many of Musk’s fans, including a bunch of people on HN, reacted with such evident glee at seeing Twitter engineers being laid off. That’s not a healthy mindset.
I hope he turns off the microservices that deliver the "Twitter is better in the app" popup, and block my screen and tell me to sign up if I try scrolling down. As a matter of fact, going web-only would be a great way to streamline
That's something that's done purely client side. Either modifying in devtools or using ublock origin's zapper tool lets you hide that. There are 3 or 4 elements needed to be erased to make everything work properly after the popup appears.
I don't know about Twitter specifically, although the fact that Musk thinks they can run the company with half the pre-sale workforce gives an idea, but when I see the size of the engineering teams at many of those companies as reported in the media (I remember for instance that the size of the Uber team is in the thousands, or that Netflix runs 1,000 microservices), I am not surprised that there may be significant bloat.
I don't understand why you're being downvoted. Even though it might not the case with all tech companies, there's the possibility there might be significant bloat with some of them.
80% bloat though? Given that Twitter had a team whose job it was to analyse the usage of running services to identify savings opportunities? I don’t believe it.
Well that's because you're starting from a premise that there are competent people that aren't named Elon Musk. From first principles you are already off to a bad start, most of the world is just flailing around waiting for him to come simplify their obvious mistakes.
Because "there might be bloat in some companies" is not a very useful, interesting, or enlightening comment. Almost every large company in the world will have some "bloat"
Also, "How does Company X have Y engineers? What do they all do?" Is one of the most common and annoying comments that constantly appear in HN threads.
It's only annoying if you don't try think about it.
If you've been in the industry long enough you have experience and you get at least a feel whether x engineers seem a lot or whether y microservices seem a lot.
Now in this case we also know that Musk has cut half the workforce so, rightly or wrongly, he has concluded that there is a lot of bloat at Twitter (not 'some company'). I can only take this as a significant piece of evidence as I don't know Twitter.
More profoundly I think this has been a consequence of the cult of 'growth'. You have to show that you're growing so you have to grow the team, you have to keep hiring and justify it by 'growth' because if you don't your company will look badly in itself and compared to peers, your IPO will not be so great, your share price will tank. Once you're down that rabbit hole you're stuck. Maybe younger engineers who have only lived professionally through this period actually believe the hype.
So I think the current wave of tech lay-offs is partly caused by this no longer being the case so that companies can be more realistic and cut bloat without fear of being seen as a lame duck.
Of course a bloated number of engineers means bloated software because all those people were busy doing something, may something not useful at all but something (including perhaps thousands of microservices...).
I've seen this kind of issue first hand. At one project, the platform team spent all of their time concocting ways to create new microservices. It's not a bad idea, but on some days I think it was just designed to sell AWS instances via Kubernetes.
That he puts microservices scare quotes says it all, really. He clearly doesn’t understand modern large scale software architectures and is falling back on his archaic, outdated, and limited experience.
This isn’t the guy you want making these kinds of decisions. Even if there is bloat in services, which there inevitably is at any large company that’s been around for a while, it’s extremely unlikely that it’s 80% of them. Especially as Twitter’s infra team have in the past already done audits of running services to try to save on operational costs.
I wouldn’t be surprised if we see a serious outage or two in the coming couple of weeks.
Microservices are simply... services. The term is stupid, but of course you get used to anything if you say it few thousand times.
I remember how long I wrote "cloud" in quotes when a decade back or so decided that a remote API from a datacenter should be called a "cloud" for some effin unknowable reason. But I got used to it.
Whether Twitter's architecture is good or not is unknowable to us. Microservice overload is an anti-pattern, not something to encourage. But in context it can make sense.
Point is a tech CEO in 2022 should not be using scare quotes around microservices.
Nor is Twitter’s infrastructure a black box, as there’ve been plenty of talks, blog posts, and other material that explains substantial portions of the fundamental architecture. We know why they adopted GraphQL as the client interface, we know why they’ve used Strato, Thrift, etc.
In fact I’d go so far as to say that the average mildly interested developer who’s been reading Twitter’s developer blogs has a much better idea how how Twitter actually works under the hood than its new CEO demanding they get rid of 80% of services.
None of this means Twitter’s architecture is above criticism of course. There’s undoubtedly legacy stuff and tech debt there. But it’s unsightly and unfair for Musk to be publicly denigrating the engineers of the company he now owns, purely because he can’t understand how things work.
As an ex-tweep, there are plenty of architectural decisions that I didn't agree with or that didn't go as well as had hoped. On the other hand, there were some amazing feats of engineering going on as well and a lot of super-talented folks. Just like any other high-visibility, high-scale system.
As much as I'd like to blame Musk's tantrums on his obvious lack of technical chops, I think it's far more likely that they're due to a combination of his piss-poor emotional intelligence and realizing that he was wrong about so many of his pre-purchase criticisms.
Now that he can actually tell that right-wing provocateurs were actually being algorithmically boosted and not suppressed, that there was no concerted "censorship" efforts (moderation is not censorship), and that the people he's been championing are full-on toxic to Twitter's viability and any chance at monetization, he'd rather burn Twitter (the service, it's most devoted users, and it's employees) down than admit he's wrong.
$44 billion say he can't burn down Twitter. If he burns down Twitter, he has to sell most of his Tesla shares. He'll fight tooth and nail to make Twitter work.
The problem is that eagerness to do a lot of work combined with incompetence & arrogance is by far the worst combination for a CEO, and I think Elon is exhibiting it with Twitter.
If he was in any way trying to save his stake in Twitter and Tesla, he wouldn't be acting out the way he is. Dude's completely divorced from reality at this point.
My wife's tinfoil-hat theory is that, after reading about the amphetamine problem at FTX, that many of the Effective Altruism cultists (Elon included) are hooked on speed and making some really horrible decisions.
20% of the services might be necessary to support home timeline rendering, but I guarantee that at least 60-75% of the remaining services are indirect dependencies.
Why is the owner of the company not reliable source?
One may not be ideologically aligned with Musk but to pretend that his tweet on the company’s inner workings is somehow less reliable than someone on HN defies logic.
An owner is well-positioned to know the truth, but also has strong incentives to mislead to serve their own purpose. (See also: all the posts about crypto exchanges recently. I'm not saying Musk is a scammer, more that there are literally examples of why owners aren't reliable sources on the front page right now.)
- Musk still has an axe to grind to justify his claim that Twitter was catastrophically mismanaged.
- He has a history of making very false statements about Twitter, including most recently that it’s the world’s biggest driver of ad traffic. That harms his credibility.
- He has been CEO for what, two weeks? And he’s suddenly discovered all this bloat stuff that others missed? Again that seems unlikely.
- This is a claim that on the face of it doesn’t pass muster, given what’s known about Twitter’s engineering approach. That should introduce more scepticism.
Sorry, but how can you possibly see his behavior and think the simpler explanation is that Musk knows anything about what he's talking about at this point as it concerns Twitter? Just because he owns it means nothing in this case - especially since he's avoided even discussing his misguided opinions with his employees?
The simpler explanation here is that he's embarrassed for making a stupid impulse purchase and now he's lashing out.
40 comments
[ 2.9 ms ] story [ 91.2 ms ] threadIs it just me though or is part of this bike shedding on micro services vs monolith. Not really enough information to determine.
Edit: I hope all their circuit breakers work correctly.
Honestly, I suspect the issue is much simpler. I think Musk detests and can’t bring himself to respect Twitter’s engineers at all, thanks to preconceptions he developed on the outside. That’s why he’s so happy to mock and trash their work in public, why he wanted to get rid of half of them without even giving them the courtesy of speaking to them, and why he believes he knows better. Expect most of those remaining to be replaced as soon as Musk finds other willing saps to take their places.
It’s the same reason so many of Musk’s fans, including a bunch of people on HN, reacted with such evident glee at seeing Twitter engineers being laid off. That’s not a healthy mindset.
Also, "How does Company X have Y engineers? What do they all do?" Is one of the most common and annoying comments that constantly appear in HN threads.
If you've been in the industry long enough you have experience and you get at least a feel whether x engineers seem a lot or whether y microservices seem a lot.
Now in this case we also know that Musk has cut half the workforce so, rightly or wrongly, he has concluded that there is a lot of bloat at Twitter (not 'some company'). I can only take this as a significant piece of evidence as I don't know Twitter.
More profoundly I think this has been a consequence of the cult of 'growth'. You have to show that you're growing so you have to grow the team, you have to keep hiring and justify it by 'growth' because if you don't your company will look badly in itself and compared to peers, your IPO will not be so great, your share price will tank. Once you're down that rabbit hole you're stuck. Maybe younger engineers who have only lived professionally through this period actually believe the hype.
So I think the current wave of tech lay-offs is partly caused by this no longer being the case so that companies can be more realistic and cut bloat without fear of being seen as a lame duck.
Of course a bloated number of engineers means bloated software because all those people were busy doing something, may something not useful at all but something (including perhaps thousands of microservices...).
But as always, 80% of what he tweets he doesn't even believe in and just does it to pump his companies' value
This isn’t the guy you want making these kinds of decisions. Even if there is bloat in services, which there inevitably is at any large company that’s been around for a while, it’s extremely unlikely that it’s 80% of them. Especially as Twitter’s infra team have in the past already done audits of running services to try to save on operational costs.
I wouldn’t be surprised if we see a serious outage or two in the coming couple of weeks.
I remember how long I wrote "cloud" in quotes when a decade back or so decided that a remote API from a datacenter should be called a "cloud" for some effin unknowable reason. But I got used to it.
Whether Twitter's architecture is good or not is unknowable to us. Microservice overload is an anti-pattern, not something to encourage. But in context it can make sense.
Nor is Twitter’s infrastructure a black box, as there’ve been plenty of talks, blog posts, and other material that explains substantial portions of the fundamental architecture. We know why they adopted GraphQL as the client interface, we know why they’ve used Strato, Thrift, etc.
In fact I’d go so far as to say that the average mildly interested developer who’s been reading Twitter’s developer blogs has a much better idea how how Twitter actually works under the hood than its new CEO demanding they get rid of 80% of services.
None of this means Twitter’s architecture is above criticism of course. There’s undoubtedly legacy stuff and tech debt there. But it’s unsightly and unfair for Musk to be publicly denigrating the engineers of the company he now owns, purely because he can’t understand how things work.
As much as I'd like to blame Musk's tantrums on his obvious lack of technical chops, I think it's far more likely that they're due to a combination of his piss-poor emotional intelligence and realizing that he was wrong about so many of his pre-purchase criticisms.
Now that he can actually tell that right-wing provocateurs were actually being algorithmically boosted and not suppressed, that there was no concerted "censorship" efforts (moderation is not censorship), and that the people he's been championing are full-on toxic to Twitter's viability and any chance at monetization, he'd rather burn Twitter (the service, it's most devoted users, and it's employees) down than admit he's wrong.
The problem is that eagerness to do a lot of work combined with incompetence & arrogance is by far the worst combination for a CEO, and I think Elon is exhibiting it with Twitter.
My wife's tinfoil-hat theory is that, after reading about the amphetamine problem at FTX, that many of the Effective Altruism cultists (Elon included) are hooked on speed and making some really horrible decisions.
That's for sure, but I don't remember many instances of tech debt or bad architectural decisions which had been solved by just shutting things down.
One may not be ideologically aligned with Musk but to pretend that his tweet on the company’s inner workings is somehow less reliable than someone on HN defies logic.
- Musk still has an axe to grind to justify his claim that Twitter was catastrophically mismanaged.
- He has a history of making very false statements about Twitter, including most recently that it’s the world’s biggest driver of ad traffic. That harms his credibility.
- He has been CEO for what, two weeks? And he’s suddenly discovered all this bloat stuff that others missed? Again that seems unlikely.
- This is a claim that on the face of it doesn’t pass muster, given what’s known about Twitter’s engineering approach. That should introduce more scepticism.
I’m gonna go with Occam’s razor here and just assume the owner of the company knows more about this than I do.
The simpler explanation here is that he's embarrassed for making a stupid impulse purchase and now he's lashing out.
One easy way to judge the reliability of this claim: https://twitter.com/search?q=2fa&src=typed_query
One of the "unneeded" microservices they just shut down was the one that let you log in with 2FA.