Rather glad speaking "truth to power" is a generally respected move at my current employer, and tussles like this - if they even happen - don't result in firings.
I'm coming at this from a different angle. It should be possible to have a heated disagreement, even in public, and not have it result in a same-day firing. In fact, some countries have it explicitly on the law books.
All this does is create an environment where people are afraid to voice opinions and address issues, e.g. because they're afraid of shoot-the-messenger. In my line of work (I make cars), dysfunctional discourse environments like that can get people killed.
Sure, and I'm a big proponent of labor. Musk is certainly not.
That said, Twitter's architecture is a Rube Goldberg contraption and Musk is clearly on a warpath to fix it, so to come out publicly and yell, "It's fine," is kind of crazy. It's not fine and you're saying you're not interested in fixing it.
Looking at the exchange, he did confirm that it's slow on Android/iOS. He claimed it's for other reasons and gave examples. He might be wrong (I haven't looked at Twitter's architecture, commentary on the technical content of the exchange would be interesting), but I wouldn't say he said "It's fine".
He said it's "net spend" and that's another way of saying, "It's fine." It's the equivalent of admitting the engine is running hot, but saying, it's not a problem with the engine, it just needs lots of oil." Frankly, I think the developer is embarrassed. There's no reason for it to be as slow as it is, other than he and the other engineers simply don't know what they're doing.
I think Musk is calling it out because he wants to see who wants to fix it and who wants to defend it.
I think you misread "Ad spend" as "net spend" in his tweet (I can't locate any using the phrase "net spend"). He mentioned that performance work correlates well with "UAM and Ad spend", which to me reads as "it's worth working more on this problem" and is indicating the willingness you're looking for.
From what I can see, he offered an opinion that the number of client requests (he gives a number of 20) and the number of internal RPCs (he gives 200) is not the primary reason the app is slow. I can't tell you whether he's right or wrong.
I mean how does he know it's not "thousands of RPC calls" -- he spent six years and never got it to perform, so he doesn't know. "You're wrong!" ? Based on what?
If I worked for six years on something and it still didn't work and someone came in and said it's "x", I would say, "I'm excited to see that data because I really want to fix this." I think I'm more upset at his attitude than anything.
In all honesty, I think what he meant was "I spent six years and failed to fix something that is fixable, and then outside consultants came in a clearly found clear problems in a couple weeks, and now I'm humiliated," but he has too big of an ego for that.
The guy's job was not specifically performance; he didn't fail at anything. Product managers drive roadmaps so even if you know there's been a performance problem for years, you might not get to invest time in fixing it. That's exactly what Eric was advocating if you read the whole thread.
Clearly these engineers have never done customer feedback for developing nations, twitter interactions clearly provide evidence for what musk said, but some engineers seem to be in denial, imho the firing is cause worthy.
I think that's misreading the exchange. He was challenging Musk on the fact that "1000 RPC calls" isn't what's happening and isn't the reason for the slowdown in his opinion, but did not deny that there are performance issues - and even cited other reasons it's slow. I don't see the denial you see.
Said engineer, said 1000 RPC calls are not the issue, and when questioned further on proving a tracepath / perf result casually exited the conversation, to me it looks like they are not ready to do better or listen to valid criticism, leading conversations on semantics is just creating noise instead of acting on signals.
Also 1000 RPC's are a very valid performance bottleneck, 1000 * latency caused by serde+ network can compounds easily based on data locality + container network orchestration, no data was provided otherwise even when asked for.
That's client-side requests. Musk was talking about internal RPC calls to microservices needed to ultimately respond to the client-side requests.
If the internal arch is heavy on microservices and you're putting together, say, 50 tweets you're at 1000 RPC calls with just 20 per tweet. It's not inconceivable.
Not to defend any of the catastrophe of management that just happened but can't both be true? Eric does say in his tweets that a lot of time is spent waiting on the network. If that is because server side, the endpoint handling a request is spamming other backend services with 100s of RPC calls it would be slow. Now, why you would call out your android dev. on why your backend is not responding fast, I dunno.
Performance has been bad forever. Twitter has had forever to fix it. They haven't. By definition, the existing developers aren't smart enough to do it. The developer here is trying to throw smoke because he's embarrassed. He should be.
It doesn't matter if it's "thousands of RPC calls" or not. Eric's response is, "It's definitely not that! It's... well... we don't know... so just spend more money on servers." That response alone would have me fire him.
Or it could be that previous management was constantly prioritizing new features instead of performance gains and now he's getting unnecessary flak for it.
The worst thing is that by firing and laying off too many people on a team with so much legacy code, they might end up scrapping the app and doing a rewrite because it'll be easier than fixing it. So inevitably, it will run faster, and people will hold it up as an example of Musk being a "genius" . When really, it's exactly what the guy suggested doing, but instead of listening they fire him and force a situation where they have to do it anyways.
> By definition, the existing developers aren't smart enough to do it.
You have no evidence to support this. Performance sucks in other countries because product leadership with their hands on the wheel of the company never prioritizes "unsexy" investments in performance and infrastructure work (same as the new guy btw). That is exactly what Eric was advocating in his thread.
Blind assumptions count for nothing. If you actually read the tweet thread Eric (the former employee) gives a summary of the trade offs made that have given Twitter bad performance and suggested removing specific features to improve perf.
Many software are like that, especially those that have been around for some time. You don't see the CEO of Microsoft spewing nonsense about Windows publicly, and then punching down on an employee for correcting him. There is no need for the circus. Technical debt is a fact of business, it can be dealt with professionally. It's not a reason to treat people badly.
The product is especially bad. The new CEO is trying to reform it. Eric is saying "spend more," which is the refuge of bad architecture. That's not the answer. And instead of privately working together to review it, he's publicly arguing. I wouldn't accept that either.
He called Musk's public statement out, because it was wrong. Twitter shouldn't be the place for this, but it's on Musk for creating such unprofessional atmosphere in the first place.
Imagine the same thing with Microsoft. We all know Windows is held together with scotch tape, not to mention that 7 years after Windows 10 they still haven't quite manage to unify the old and the new interfaces, not that Windows 11's new interfaces are any good. What a mess. Except it's not, it's business as usual. Big software projects are messy and hard to manage, but these issues are unimportant when you get the bigger picture right. For Windows the bigger picture is that it runs all your application. This is what makes Windows a good product, the rest is details and can be fixed over time. Preferably in a professional manner.
How is Musk trying to 'reform' it when since he's owned it Twitter has had worse performance than ever, degraded features due to his bull-headed decisions and is literally burning money because he's chasing advertisers?
And the only one that started a public argument was Musk by making an incorrect claim and defacto blaming the engineers for something he clearly doesn't understand. He's trying to pull another Paypal and it's blowing up in his face now like it did then.
Eric didn’t defend the entire architecture though. Space Karen made a specific claim (“>1000 RPCs to render the time line”) and Eric refuted just that one claim.
No he didn’t. Eric essentially said “the clients don’t make >1000 RPCs to build the time line”, a direct refutation of Space Karen’s original assertion.
Yes, but he went on to say that it is truly slow and that it needs more spend, which is another way of saying, "You're wrong! It's not that! ... but I have no idea what it is," which isn't a very good response.
That’s a pretty uncharitable interpretation. I read it as “You’re wrong” (which is correct - Space Karen was wrong in his claim about why things were slow) “and we should figure out why, which will take time & effort (i.e. investment).”
You could be right. But how do you know the consultants Musk hired are wrong? How does Eric know? Maybe there was a situation found where that did, in fact, happen. Certainly with a large number of microservices, it could happen. Maybe Musk is saying "edge case" and Eric is saying "on average." Personally I feel outside consultants could have very well found problems that the internal team did not see.
What everyone agrees on is that Twitter's architecture is deeply broken. Eric's been there six years, so he's probably at least a little culpable in that. If I'm Eric, and was part of a team that fucked up that badly for six years, I certainly wouldn't argue back with that much confidence.
It’s not that hard to aggregate multiple APIs to a single request.
I guess it’s not that easy. Space Karen did just tweet the first thing he heard from his stuff. And it’s more clearly he doesn’t know how to do PR by his own.
It’s different to apologise for bad performance or blame implicitly the developers.
Good for Eric. What kind of CEO trashes his own employees (and the product which he now owns) in public like this? It's a good thing Musk wants to reduce headcount because it's hard to imagine anyone taking an offer from Twitter at this point.
If there's anything about Musk's leadership style I find the most... confusing? it's that he doesn't seem to like to be challenged. I would consider the people most willing to productively call me out are my most valuable assets.
This is a very interesting aspect of the very visible way that Musk has taken over Twitter. For example, not knowing his style, one would assume that decisions like making full self-driving use only camera sensors was discussed in depth and contrary points of view were carefully considered. That might have happened. But based on what we all now see, it also might not have happened at all. The same goes for Starship. Is it a breakthrough innovation? Or is it an idiosyncratic founder's Spruce Goose?
Tesla's board has had well-known independence issues for a while. Musk's insanity while running Twitter adds credibility to Musk's detractors at Tesla tho, which is probably why Tesla's been plunging lately
> because it's hard to imagine anyone taking an offer from Twitter at this point
That's what increased compensation is for.
Except in this case, Elon has both pushed for cutting costs (negating the ability to pay increased compensation) and also pushed for a much worse work-life balance to the point that increased compensation may still result in worse $/hr than another tech job.
Also, the equity investors are all solidly underwater. The debt holders are under water by maybe half. So anyone with an incentive package is last in line behind the creditors and the equity investors.
Dude just said Elon was wrong. You'd think the self-declared free-speech absolutist would've been happy to publicly debate him in the marketplace of ideas (or however that old chestnut goes). :)
Musk didn't delete his Twitter account, so he's still able to debate in the public Town Square. The free-speech exhortation doesn't apply to his job, apparently. No winners here.
Eric Frohnhoefer may not have a job at Twitter anymore, but him fact checking dear leader constantly spewing nonsense about things he doesn't understand was a sweet moment.
Elon was talking about internal RPCs. Eric was talking about client side requests, as though that was a dunk. Eric was both wrong and attempted to insult his boss in public at the same time.
Take into account that between the two of them Eric is the expert.
It was Musk who failed to keep things professional. His first post was literally humiliating his own employees for writing bad code. It's his own company, he may not have been around, but it's his people now. Even if Twitter's Android client is not the best, playing the blame game is callous and petty. There is no point to trying to show that you are better than your employees either. It's the opposite of what a functional CEO should do.
"I was told ~1200 RPCs independently by several engineers at Twitter, which matches # of microservices. The ex-employee is wrong.
Same app in US takes ~2 secs to refresh (too long), but ~20 secs in India, due to bad batching/verbose comms. Actually useful data transferred is low."
There is no way to tell if this is right or wrong, we are not engineers at Twitter, and I suspect that no engineer there would dare to correct him. I'm pretty sure Twitter has an API gateway of some sort, it doesn't make sense that it's a batching issue.
But it doesn't matter who is right or wrong. The unprofessional atmosphere, the callousness, the one-upmanship is the problem. He is the CEO for God's sake. This is a PR nightmare. I know that it worked for Trump, but it isn't working even for him anymore.
I honestly don't understand what remote procedure calls have to do with the client. Surely the clients wouldn't interact in such a way with the microservices, there would be some sort of API gateway, maybe even one for each platform or service, like the backend for frontend pattern.
It's probably more complicated than this, and you can't exactly put it into a few words, but he tried, and it didn't make sense, but he still managed to call his employees idiots.
It was callous and petty, and I can't imagine how frayed the nerves are of current employees. Almost every higher-up I've seen come into a company has done this to some degree, and it's always annoying, but to do it actually on the product to 100mm+ followers is a new level of cruel and spiteful.
> An interesting thing about this claim is that not only is the implication wrong, Twitter probably has better evidence of its wrongness than any other company in its size class could have.
> There are very few companies that have a better distributed tracing setup w.r.t. getting actionable insights on the backend and the ones that have a better setup are much larger (Google, FB, etc.)
> Twitter client tracing also punches above its weight.
> Of course, the key people who did that work left or got laid off, but it's clear from the data that, if you're looking at why Twitter is so slow in, e.g., India, Uganda, etc., esp. on slow devices, tail latency comes from the network due to unreasonably large payloads + client.
> When leadership wanted to drive growth in India, they were into "visionary" stuff, e.g., "add features cricket watchers would like" when the app was unusably slow [on] a low-end device and the client performance team had been disbanded.
Elon apologizes for the poorly written app and argues that it needs to be completely rewritten.
A developer unnecessarily jumps in front of the bullet to take the blame and say that Elon is wrong. It is poorly written in a different and vague set of ways. And the only solution is a complete rewrite: https://twitter.com/EricFrohnhoefer/status/15919684638491525...
Not defending Musk here, but this is a learning moment in managing up - it doesn't matter where you work, you probably shouldn't be publicly ridiculing your boss, especially for being technically incorrect. Especially when that boss was trying to hand you a blank check to give you what you want.
So much for Musk's grand vision of a town square for everyone to be heard. He can't even handle 1 "maybe-critical" remark without firing the guy. Sheesh.
I believe Elon did what people would call a textbook cancelling. That is, he harassed them over Twitter and then caused them to lose their job due to making false statements.
LOL. This guy has a 9-month old account and has posted dozens of comments praising or defending Musk. It looks like half of his comments are about Musk. His very first comment was praise of Thiel. Hilarious.
Every sufficiently wealthy and famous person hires a publicity team to manage their public image. The service includes paying interns to go around social media praising and/or defending their client. Some people will do anything for a paycheque, including debasing themselves for an obnoxious billionaire.
Sure it was suicidal of an employee to call out their boss in public, but even more destructive for a CEO to spout nonsense without researching first, because it makes him look like an incompetent ninny.
EDIT: Steve Jobs wouldn't have whined like Musk, if anything he would have denied the problem existed while choking-throats behind the scenes to fix it (unless the "bad thing" happened live during the Apple developer conference...).
Elon needs Eric one hell of a lot more than Eric needs Elon. Eric will have a new job within days. Elon will have a developer with six years deep knowledge and experience with Twitter code… in about six years.
80 comments
[ 3.3 ms ] story [ 153 ms ] threadRather glad speaking "truth to power" is a generally respected move at my current employer, and tussles like this - if they even happen - don't result in firings.
All this does is create an environment where people are afraid to voice opinions and address issues, e.g. because they're afraid of shoot-the-messenger. In my line of work (I make cars), dysfunctional discourse environments like that can get people killed.
That said, Twitter's architecture is a Rube Goldberg contraption and Musk is clearly on a warpath to fix it, so to come out publicly and yell, "It's fine," is kind of crazy. It's not fine and you're saying you're not interested in fixing it.
I think Musk is calling it out because he wants to see who wants to fix it and who wants to defend it.
From what I can see, he offered an opinion that the number of client requests (he gives a number of 20) and the number of internal RPCs (he gives 200) is not the primary reason the app is slow. I can't tell you whether he's right or wrong.
If I worked for six years on something and it still didn't work and someone came in and said it's "x", I would say, "I'm excited to see that data because I really want to fix this." I think I'm more upset at his attitude than anything.
In all honesty, I think what he meant was "I spent six years and failed to fix something that is fixable, and then outside consultants came in a clearly found clear problems in a couple weeks, and now I'm humiliated," but he has too big of an ego for that.
Also 1000 RPC's are a very valid performance bottleneck, 1000 * latency caused by serde+ network can compounds easily based on data locality + container network orchestration, no data was provided otherwise even when asked for.
If the internal arch is heavy on microservices and you're putting together, say, 50 tweets you're at 1000 RPC calls with just 20 per tweet. It's not inconceivable.
The developer puts the real number at 200.
You have no evidence to support this. Performance sucks in other countries because product leadership with their hands on the wheel of the company never prioritizes "unsexy" investments in performance and infrastructure work (same as the new guy btw). That is exactly what Eric was advocating in his thread.
The product is especially bad. The new CEO is trying to reform it. Eric is saying "spend more," which is the refuge of bad architecture. That's not the answer. And instead of privately working together to review it, he's publicly arguing. I wouldn't accept that either.
Imagine the same thing with Microsoft. We all know Windows is held together with scotch tape, not to mention that 7 years after Windows 10 they still haven't quite manage to unify the old and the new interfaces, not that Windows 11's new interfaces are any good. What a mess. Except it's not, it's business as usual. Big software projects are messy and hard to manage, but these issues are unimportant when you get the bigger picture right. For Windows the bigger picture is that it runs all your application. This is what makes Windows a good product, the rest is details and can be fixed over time. Preferably in a professional manner.
And the only one that started a public argument was Musk by making an incorrect claim and defacto blaming the engineers for something he clearly doesn't understand. He's trying to pull another Paypal and it's blowing up in his face now like it did then.
What everyone agrees on is that Twitter's architecture is deeply broken. Eric's been there six years, so he's probably at least a little culpable in that. If I'm Eric, and was part of a team that fucked up that badly for six years, I certainly wouldn't argue back with that much confidence.
I guess it’s not that easy. Space Karen did just tweet the first thing he heard from his stuff. And it’s more clearly he doesn’t know how to do PR by his own.
It’s different to apologise for bad performance or blame implicitly the developers.
That's what increased compensation is for.
Except in this case, Elon has both pushed for cutting costs (negating the ability to pay increased compensation) and also pushed for a much worse work-life balance to the point that increased compensation may still result in worse $/hr than another tech job.
A: One that doesn't want to work there and is looking for attention on the way out.
There is only one way this would ever go. Would you, as the boss, keep such a person in your employ?
It was Musk who failed to keep things professional. His first post was literally humiliating his own employees for writing bad code. It's his own company, he may not have been around, but it's his people now. Even if Twitter's Android client is not the best, playing the blame game is callous and petty. There is no point to trying to show that you are better than your employees either. It's the opposite of what a functional CEO should do.
Same app in US takes ~2 secs to refresh (too long), but ~20 secs in India, due to bad batching/verbose comms. Actually useful data transferred is low."
https://twitter.com/elonmusk/status/1592176202873085952
But it doesn't matter who is right or wrong. The unprofessional atmosphere, the callousness, the one-upmanship is the problem. He is the CEO for God's sake. This is a PR nightmare. I know that it worked for Trump, but it isn't working even for him anymore.
I honestly don't understand what remote procedure calls have to do with the client. Surely the clients wouldn't interact in such a way with the microservices, there would be some sort of API gateway, maybe even one for each platform or service, like the backend for frontend pattern.
It's probably more complicated than this, and you can't exactly put it into a few words, but he tried, and it didn't make sense, but he still managed to call his employees idiots.
https://twitter.com/danluu/status/1591963269077819392
> An interesting thing about this claim is that not only is the implication wrong, Twitter probably has better evidence of its wrongness than any other company in its size class could have.
> There are very few companies that have a better distributed tracing setup w.r.t. getting actionable insights on the backend and the ones that have a better setup are much larger (Google, FB, etc.)
> Twitter client tracing also punches above its weight.
> https://twitter.com/danluu/status/1267069847428411392
> Of course, the key people who did that work left or got laid off, but it's clear from the data that, if you're looking at why Twitter is so slow in, e.g., India, Uganda, etc., esp. on slow devices, tail latency comes from the network due to unreasonably large payloads + client.
> When leadership wanted to drive growth in India, they were into "visionary" stuff, e.g., "add features cricket watchers would like" when the app was unusably slow [on] a low-end device and the client performance team had been disbanded.
> https://twitter.com/danluu/status/1591507609651408896
A developer unnecessarily jumps in front of the bullet to take the blame and say that Elon is wrong. It is poorly written in a different and vague set of ways. And the only solution is a complete rewrite: https://twitter.com/EricFrohnhoefer/status/15919684638491525...
Not defending Musk here, but this is a learning moment in managing up - it doesn't matter where you work, you probably shouldn't be publicly ridiculing your boss, especially for being technically incorrect. Especially when that boss was trying to hand you a blank check to give you what you want.
Anyone that openly slaps down that asshole Musk is doing a public service, in my book. I hope more employees exit like this.
EDIT: Steve Jobs wouldn't have whined like Musk, if anything he would have denied the problem existed while choking-throats behind the scenes to fix it (unless the "bad thing" happened live during the Apple developer conference...).