Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
>Mr. Musk is also considering shuttering one of Twitter’s three main U.S. data centers, a location known as SMF1 in Sacramento, which is used to store information needed to run the social media site, four people with knowledge of the effort said. If the data center in Sacramento is taken offline, it will leave the company with data centers in Atlanta and Portland, Ore., with potentially less backup computing capacity in case something fails.
There are currently three data centers, hosting all of the real-time and most of the batch production load. Ad-hoc load and some production batch load was migrating to GCP, but this was being significantly curtailed in the months before the acquisition closed because it turned out to be very expensive, much more so than anticipated and more than the equivalent workloads had cost in the data centers.
What did you make of Mudge's report regarding resiliency of data-centers?
> Insufficient data center redundancy, 59 without a plan to cold-boot or
recover from even minor overlapping data center failure, raising the risk of
a brief outage to that of a catastrophic and existential risk for Twitter's
survival.
Thanks! There was a lot in that and I didn't follow it closely. Honestly that always confused me about the dc failing prep. Like I get that it would be a good idea to have a documented plan but at the same time, we did lose a whole DC and figured out how to recover(not that it was that easy) Im not sure having a cold boot prep plan would have helped that much. The things that failed that I dealt with personally during that event, I don't know if I would have foreseen. The site went on afterwards so it didn't seem to be existential.
Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
Right. It’s not like if all the engineers walk out it’s suddenly going to fall over. It’s that when an issue comes up there may not be the expertise on hand to fix it. So the remaining twitter engineers should expect a rough time over the coming months.
There must be some plan for this, though, right? If it were me and I were trying to do more with less I'd plan to cut some features (for instance, are Spaces that critical to the operation? Because it seems like running them would be demanding) and try to migrate things to managed services to reduce the operational load.
I don't have a DR plan for "mad billionaire buys the company", no, but I do agree with the OOM reaper logic - I start cutting the services that use the most resources that are least necessary.
I don't mean that they had one sitting around, but anyone who's stayed with the company has to find this their most urgent priority, I would think, between the focus on cutting operational costs and the simple fact that they have far fewer people around.
Well, that was a bit of a joke, but I have been around the block a few times and been on one side of a merger or other, and gone public to private and back (as an IC/soldier, never been management). The Twitter acquisition is the craziest thing I've ever seen; there's a reason it's big news.
These kinds of things have formal courtship type periods and legal filings that take months, and then they happen, and then executives plan on rolling out layoffs and cost cutting measures over years, not days or weeks, like the "plan" seems to be here. This really is madness, and I can't blame people for thinking the site might just wobble and collapse at some point, though as the author goes into in very nice detail, we do try and build resilient systems. I'm certainly curious how it all plays out, but you won't catch me making any guesses.
This kind of schedule and chaos looks more familiar for acquisitions of a small company I think. I guess it is strange for it to happen to a company of Twitter's size, but I think the big-news aspect of it has more to do with how much journalists use Twitter.
> This kind of schedule and chaos looks more familiar for acquisitions of a small company
That's certainly possible and I wouldn't know what that would look like, and yep, in theory there are lots of financial rules and regulations regarding public companies that Musk made a bit of a hash of.
I can't agree that Twitter's journalist demographic was mostly responsible for this being big news, though I'm sure that plays a part. Musk brings the circus with him wherever he goes, but this was nuts even for him. And any $44 billion tech buyout would be news without those factors.
> I guess it is strange for it to happen to a company of Twitter's size, but I think the big-news aspect of it has more to do with how much journalists use Twitter.
I think you hit on the the main point: the reason this is constantly in the mainstream media is because the mainstream media so heavily relied on/still relies on twitter[1].
If this were any other large company imploding in a Musk acquisition we'd see fewer stories about it because it affects journalists less.
--------
[1] It's going to be especially funny if journalists complain that "#LearnToCode" type of tweeters aren't getting banned, and get told in response to go off and create their own twitter.
No matter where in the political spectrum one may lie, it's always satisfying to see a group being fed their own lines back.
You assume that small issues are ever prioritized in the first place. My experience with large tech organizations is that their priories are driven by internal culture, not best practices, just like any other company.
Although I haven't worked at Twitter, if it is anything like the places I have worked at, the small issues get ignored because nobody makes their career on things that don't show up on investor / shareholder / upper management radars. If it can get papered over so it ends up being someone else's problem in the future, what actually gets worked on is flashier things like redesigning the timeline AGAIN so users spend more time on the website or (in the case that caused me to work for small startups again) spend three weeks deciding on a super trivial design decision with 12 stake holders.
I found it really interesting that one of the main Iceland instances is being run on a Raspberry Pi out of some student's living room in Sweden. Over 500 users at the moment.
https://types.pl/@tritlo/109383888427885539
First, Mastodon isn’t the only option - it’s not like both couldn’t fail just as both AIM and MySpace did.
Second, Mastodon runs on open protocols. That has good and bad points - for example, it won’t grow as quickly as a project with huge corporate backing - but it does mean that there’s a more direct link between the community and its longevity. Twitter isn’t just flailing because Musk is doing management by bong rip but also because he’s desperately trying to get out of a financial hole. Open source projects have different kinds of financial challenges but they’re never on the hook trying to fill a hole measured in billions of dollars, either. Given the number of communities older than Twitter I’d say it’s far from proven that Twitter will outlast anything.
why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate
- completely dead with ssl invalid certificates with expired domains.
So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above.
There is no monetary incentive to keep a mastodon instance running and we both know that begging for donations doesn't scale.
There's a gigantic flood of new users (i.e. literally multiplying the userbase) in the past few weeks, so yeah, a lot of servers are restricting signups to cope.
I don't think "begging for donations" needs to scale, if instances get too large to keep running then smaller instances should (and do) split off, they can still talk to each other after all.
software, like everything, is subject to laws of physics
I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.)
Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" and change. For the preservation of their own self-interests, of course; much like the company that makes physical objects too robust and runs out of customers, planned obsolescence and the desire to change things and justify it so they can be paid to do something are still there.
It's possible to make things which last. Unfortunately, much of the time, other economic considerations preclude that.
Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the specs of the target platform change but a hard-working compiler engineer has done the work to make sure your software gracefully uses the new features (for example, a compiler optimizing using AVX instructions), your software should still function.
The fact that most software doesn't continue to function even on the same platform, and on the same hardware, is a massive indictment of the software industry's standard practices.
You're forgetting that software on it's own is basically useless. In order for it to provide value, it has to be operated by a physical machine. All running software is physical, with spinning disks and mechanical relays, electrons being pushed back and forth, and photons flying around. Twitter is not a piece of software, it's a complicated physical system. software is an abstraction.
Unfortunately it’s not this simple because most non-trivial software is written with a dependency tree, every node of which may discover vulnerabilities (or performance problems) which, when patched, trigger update cascades in this tree.
Complex software has complex failure modes though.
An application running on a single platform, self-contained and with some basic failovers such as redundancy (2+ machines running the same application), etc. should have ridiculous uptimes.
A distributed and complex system with interdependent components, under variable load, with different capacities for subsystems running across some thousands of machines will, inevitably, encounter some unforeseen state which degrades the system as a whole. It can be a small component that cascades a failure mode in an unexpected way or can be a big component failing spectacularly due to a bug, or race condition, or a multitude of other issues that are not entirely predictable and guarded against at the time of writing such software.
The latter is what has "wear and tear", it's not one software, it's a whole system of software communicating with each other in varying states of hardware decay, you can design and build it to be resilient against a multitude of predictable issues but you can never expect that it will run perfectly fine unattended.
Even if the entirety of Twitter.com were mathematically proven correct, it still would run on servers that are made of physical bits that are subject to entropy.
If software ran without side effects, perhaps. But it doesn't. Databases grow, files are uploaded, logs pile on, messages and events propagates and filesystems fill up. This is why entropy matters.
Exactly. Tiny memory leaks in seldom called functions can also cause slow degradation over time. People wonder why a simple restart seems to 'fix a boatload of problems' but this is often the reason why.
> we don’t think so. the prod incident we heard about involved someone making an ill-advised choice to reactivate a large account, causing a huge load on the social-graph system, on the night before a prolonged high-traffic event.
Don't forget that users have been trained to be incredibly fault tolerant as a result of how flaky general software can be. Now that cars are having BSOD that tolerance may reach new levels or just evaporate.
It’s possible to make things that last if you are in total control of the whole stack, including hardware.
Embedded systems that still do their job after 30 years do exist but they live in isolation in a specific and controlled environment, and are built for a limited, unchanging task.
On the other hand, complex web software is build on layer upon layers that are not in Twitter’s complete control.
Hardware change regularly, requiring changes at the lower levels of an OS, inducing potential changes in behaviour, performance, which require adaptation as a consequence.
And that’s before considering security, eternally moving goalposts. Not just at the OS or network level, but also at the business level.
Twitter and al are not living in a locked down context, they live in the messy world of human interactions and that alone requires constant tweaking.
So yes, a binary is more like a mathematical construct and by itself it won’t rot, but if the world around that binary changes, you need to change the binary as well, and for that you need maintenance. The amount required depends on the complexity, brittleness and how well your stack is engineered, but implying it’s a con is a bit extreme.
> I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.)
Sure, those binaries might work the same when executed. Although the probability of that is never 100%, but as you pointed out, the rules of arithmetic aren’t expected to change any time soon. That’s correct. Unfortunately software does not exist in its own micro-verse, it’s subject to the laws of physics acting on the machines it’s running on. So while you might be able to write scripts that work decades later, it’s much harder to ensure those scripts consistently run for decades. RAM chips, CPUs, and everything in between are guaranteed to eventually fail if left running unsupervised in perpetuity. Entropy raises with complexity. At Twitter’s scale, to run a software service you need globally distributed cloud infrastructure. They likely have hundreds of services, deployed to many hardware instances distributed across the globe. Twitter isn’t 1 script running 1 time producing a single result. It’s hundreds if not thousands of systems interacting with one another across many physical machines. Layers of redundancy help, but ultimately cascading failures are a mathematical certainty. Many would argue the best strategy to reduce downtime on these systems is to actually optimize for low recovery time when you do fail.
Software is also bound to the world in other ways. Similarly to how most business processes, products and even more generally, tools, change over time, so too do the requirements placed on software systems made to facilitate or automate these things.
Ultimately the only way to escape the maintenance cost of software is to stop running it. The longer you leave a software system running, the more likely it will eventually stop.
Computation is literally bound by entropy. Math has no such limitations unless you explicitly define them.
I thoroughly recommend researching entropy as it regards to e.g. information theory, systems engineering and even (perhaps especially) to machine learning.
Computation is ultimately about what we can compute _in this universe_ and the forward flow of time is an emergent property from the universe’s innate entropic guarantees.
Time is “pre-sorted” for us thanks to entropy, enabling us to define algorithmic complexities over the time domain in the first place.
Turning it off and then back on again probably fixes the issue. There’s very unlikely a grand ticking time bomb just waiting to bring it all down. Recycling servers will probably keep it running.
> Turning it off and then back on again probably fixes the issue.
Turning a large scale system entirely off and on is never simple. Invariably you’ll run into some kind of circular dependency that must be manually investigated. And even tracking those down becomes tricky.
Classic examples are things like DNS, service locators, or authentication systems. And large tech companies are notorious for NIH-syndrome for all of those.
There’s so much redundancy built into modern distributed systems that you can reliably bounce a VM without issue. You can reliably roll bounce a series of VMs.
Twitter doesn’t have unique scale problems by todays standards.
Thanks I agree, I think small issues can build up over time, itll be interesting to see what happens. Wish I could see the future post mortem docs from the outages
I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.
> Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
Did we read the same article, because it seemed clear that all that resilience work was done when the company was public shareholder owned, not under the new private owner.
I read the article, I wanted to use the space to point out that Elon's fortunate that it has been done, and also that the technology infrastructure things he needs to focus on aren't impossibly challenging to do, if people are expecting it to be.
So operating Twitter with 80% fewer engineers isn't the voyeuristic suicide that many of us are hoping to be amused by.
The problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twitter.
He's going to sell Tesla shares to pay off his debts, he's knocked off $4 billion already, he'll be able to keep doing that, make offers to buyout stakes of Twitter from people that really don't want to be involved in this shit show. Twitter will operate at lower costs, and also not be profitable for him given current information. That's financially fine.
The banks probably took convertible debt at high interest rates, but who cares about high interest rates when it's only been one month. He probably has a clause for accelerated payments, because its debt, not shares yet.
(after a certain amount of time, or upon default, the debt would convert to shares. but it can be much more complicated, and onerous than that too. lending can be fun.)
Musk has been in a similar position before when he took over Tesla. At that point it was struggling to make the Roadster, they were facing having a marginal cost of manufacturing a car higher than the sales price.
Musk took over and made cuts. I don't think anyone could argue that those cuts were at the cost of Tesla's long-term growth. They were needed because of an imminent liquidity problem. Tesla has hired great people since then.
I wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load.
This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.
Interestingly, my reports on racist posts are being accepted much more often than they were before, so this is in fact not happening.
The site was also consumed in horrific discourse for the last week (topic being "is making chili for your neighbor racist, ableist and oppressing autistic people?") so all in all it seems to behaving as usual.
Exactly what kind of views do you think moderates were worried about not being tolerated? Going by the definition of the term it seems unlikely that anyone, much less millions, was previously deterred by policy.
He wasn’t banned (only suspended for a few hours) and that was directed at a single trans person. That last part is almost certainly the reason why he got that warning unlike the many people who have generically expressed similar beliefs.
Their previous CEO said they saw on the order of 500k bot signup attempts daily, and the reports last week specifically mentioned layoffs affecting those teams and things like reporting. Put that together and I’d be careful about comparing numbers before and after this month since they’re almost certainly not measuring the same thing.
Interesting how this summer all we heard from Musk was how Twitter was overflowing with bots and spam, but now that he runs the place he sure spends a lot of time bragging about all-time high "user" counts.
Catastrophe tourism. Also, contrary to Elon’s shitty graph with cut off y-axis (can we stop doing that?), that’s like a 1% change around a major American event.
Very interesting article. But the constant misuse of “to” for “too” kept throwing me out of reading mode. I wonder why my brain does that instead of reading past it since I know what is actually meant.
I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.
Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own.
And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.
I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.
I will admit to this. I was so angry at how Musk treated his employees, and I took them at face value when they predicted outages. I still think that we will see outages in the coming weeks and months, but probably not in spectacular fashion.
A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).
It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees.
It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail.
> put their reputation on the line saying Twitter wouldn't survive the weekend.
Was that a real thing those people were saying?
Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover.
So it wasn't unreasonable to think that Twitter would begin experiencing problems.
I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way.
It comes off as a kind of desperation, as
though they need Elon to fail.
You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face!
If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy.
But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!
But the parent poster characterizes those disliking Elon Musk as suffering from "psychosis" and "desperation."
It's a an unfortunately common, passive-aggressive ad hominem tactic you see on the internet and elsewhere. "People don't like thing or person XYZ? Oh, they must be mentally ill and losers who spend their whole lives obsessing over that thing/person"
Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things going wrong.
More so at companies like Twitter where they never really reach a steady state. You constantly have large fluctuations in system stress e.g. World Cup, Trump rejoining etc.
Just curious, among the critical stuff just holding on, was there also a whole pile of departments and teams doing work that could cease tomorrow and the company would blink and move on?
My experience is both exist at the same time because the leadership teams don't actually know what core business is or are busy building empires and resumes.
Just curious, among the critical stuff just
holding on, was there also a whole pile of
departments and teams doing work that could
cease tomorrow and the company would blink
and move on?
I get what you're saying. For any given team with a public-facing product you generally have perhaps 20% of the staff keeping things running and the other 80% of the staff is working on new features, reports, enhancements, customer support, whatever. You could eliminate them and while it would diminish the company in various ways, the other 20% could certainly keep the lights on.
However it's worth noting that's not what happened at Twitter; there were very specific and explicit reports that the "keep things running" teams were hit just as hard by layoffs/resignations as the other departments.
So there was real justifiable concern there.
There's also a lot of things that can go wrong during sloppy and abrupt handovers. Like... you fired the guy who manages the domain renewals. In the chaos of transition nobody picks this role up. One day 18 months later you realize "twitter.com" has expired. Or whatever. Even if the remaining staff is sufficient to keep things running, there are thousands of these little process interruptions.
What is with all the people this year who insist that because something doesn't happen right away, it can never happen?
The only prediction anyone made was that the World Cup was historically a period of very high load, so if something was going to go wrong soon, the weekend would be the first vulnerable time, and the World Cup finals will be the next.
It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing.
Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).
I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office.
e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.
Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory, and all the other ones have said exactly the same thing to me as you are now. Sure, I'm a secular-heretic for wanting to see evidence for extraordinary assertions from people with obvious ulterior motives, rather than taking them on faith.
So far, the conspiracy theorists have had a poor track record as far as I can see. That said, I don't engage or try to debate my friends on it, they have always become extremely upset by any debate on such topics. And they're not hurting anybody so my philosophy is just to leave them to their faith. I don't go around challenging people to prove the existence of God whenever they make a religious Facebook post either, I'm not that kind of asshole.
> Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory
As you can read in the Republican-written Senate Intelligence Committee report, everything about Russiagate did in fact happen and it was 100% real. (except for the Steele dossier/pee tape)
This essentially comes down to Trump realizing that if you say all your crimes in public, nobody will think it's a crime and it won't be reported as one, because you don't "seem like you have something to hide".
I might be basically be picking at it with what I am about to say next, but was it not admitted that 'Steele dossier' itself was a rather elaborate disinformation campaign[1] as noted by Fiona Hill? I really don't be want to be this guy, but if we are going to have that kind of discussion here, we might as well discuss current state of knowledge.
edit: All this is relevant, because that dossier itself was the basis for 'Russiagate'. I am ok with a rational counter-argument.
> As you can read in the Republican-written Senate Intelligence Committee report, everything about Russiagate did in fact happen and it was 100% real. (except for the Steele dossier/pee tape)
The conspiracy theory is that Trump colluded or conspired with Putin to hack the election. No evidence to support this has ever been produced. I know that may be difficult to accept, and I ascribe no malice to those who genuinely believe conspiracy theories, they are a powerful psychological trap.
When Trump told Russia to release his opponent's emails they'd hacked, and they did release them afterwards on the same day, he got away with it because he just did it in public on stage, and so cynics can't accept it because it's too obvious - it wasn't done on an encrypted Signal chat in a smoky backroom, so it must not be a conspiracy, so it must've been a coincidence.
Plus his campaign manager was working for him for free and instead being paid by a GRU agent for their targeting data.
You can try denying the Senate/Mueller reports on this, but you should read them, since they do say this.
The joke he made at a political rally about Russia releasing Clinton's emails if they had them? That simply is not not evidence that Trump colluded with Putin to hack the election. Calling that evidence is like a climate denier calling an unseasonal snowfall evidence that global warming is false. And that's not what the misinformation peddlers at the time were claiming to be the evidence, either. Remember Adam Schiff's repeated false claims of "ample evidence" that he could not share but alleged were shown to the House Intelligence Committee that he chaired? That was the kind of inflammatory misinformation that drove the whole conspiracy theory.
This is how conspiracy theories hook people, they tell people what they desperately want to be true, and they falsely claim there is this vast amount of solid, air-tight evidence, and it will all soon come out. That gets people invested, hooked, convinced that it is true. When things eventually get disproven much later and it turns out there is no such evidence forthcoming, the believers are so invested in it that they are incapable of accepting the reality, so they cling to whatever flotsam they possibly can -- "Trump did collude with Putin to hack the election, therefore this joke at a campaign rally must be a message to Putin, therefore it is irrefutable evidence that Trump colluded with Putin to hack the election". It works on probably a subconscious level, it is a very difficult to step back and see this when you are in the hole, let alone dig yourself out of it.
No report into this including the vaunted Mueller Report has ever produced any evidence that Trump colluded or conspired with Putin to hack the election. I have read them. Mueller Report On The Investigation Into Russian Interference In The 2016 Presidential Election, Volume I, p. 181, second paragraph:
For that reason, this Office’s focus in resolving the question of joint criminal liability was on conspiracy as defined in federal law, not the commonly discussed term “collusion.” The Office considered in particular whether contacts between Trump Campaign officials and Russia-linked individuals could trigger liability for the crime of conspiracy—either under statutes that have their own conspiracy language (e.g., 18 U.S.C. §§ 1349, 1951(a)), or under the general conspiracy statute (18 U.S.C. § 371). The investigation did not establish that the contacts described in Volume I, Section IV, supra, amounted to an agreement to commit any substantive violation of federal criminal law—including foreign-influence and campaign-finance laws, both of which are discussed further below. The Office therefore did not charge any individual associated with the Trump Campaign with conspiracy to commit a federal offense arising from Russia contacts, either under a specific statute or under Section 371’s offenses clause.
Citizens United case means that, by design, there will be no evidence of money transfers from foreign powers to US elections.
so the idea we can 'find no evidence' is because people rigged the system so there would be no evidence.
Democrats could start taking money directly from Xi Jingping or from some communist in Venezuela and we would never know about it. Republicans can take money from the Saudis and we would never know about it. This is the reality of modern politics in the US.
infinite invisible money and zero evidence that anything happened, by design
Right, if you're already deep in the hole and all-in on a conspiracy theory, there are endless ways you can explain away holes in the theory and turn conjecture or coincidence into hard evidence. Even the lack of evidence can be evidence to people taken in by conspiracy theories.
Same as how China secretly paid the liberals to create the global warming hoax, that's another one along the same lines as the Russia collusion one that some people actually believe.
Some of our friendships are not based merely on political affiliation. This is not a shot, but please understand that not everyone is automatically looking for a clan and an enemy to bond over ( even though it is not uncommon and perfectly rational thing to do based on our evolutionary path alone ).
Not to search very far, me and my coworkers discuss politics ( and we do have somewhat divergent views ). The bubble only exists if one does not talk with other people. In other words, other people being like-minded is not an automatic pre-requisite.
I actually find this fascinating and and am not taking sides. If Jan 6th was literally an attempt to overthrow the government, why hasn’t Trump been convicted of treason? I thought swift action would be taken if the slant in the stories news media have reported is accurate.
But more importantly, why can’t you be friends with people who have different politics? Honestly I’m not sure what word best describes a world where everyone has to think the same to get along, but I have to think it’s far from ideal.
The bubble where the cronies who helped plan it were huddled together in the Willard to watch the show put on by the people they directly interfaced with and who were equipped to breach the building.
I personally have somewhat mixed feelings. I do enjoy a good dumpster fire like any good internet denizen ( and I won't deny that there is an odd level of.. excitement maybe.. that one major social platform may go down ). I have no love for social media as some of my posts may indicate and for a variety of reasons including the amount of oxygen this creation takes and the amount of confusion it causes in general population, it would not be the worst thing ever.
That said, the sheer amount of weird emotion attached to this particular billionaire is just perplexing. Only few years ago he could do no wrong. Today, it seems, he is being ganged up on including by some of his biggest internet fans. And media oblige for whatever reason. I dislike billionaires, but I dislike them as a caste in our societies. I don't see Elon as a savior of mankind or the devil, but those emotions that swirl around him are made to make him look like one or the other.
Somehow, one is supposed to pick sides.
Trump seems to create the same emotions. It is truly fascinating.
I was about to say that you're talking about two different problems. But, no, what's happening to Musk is the same thing that happened to Trump. Their inner circles knew how rotten they both were, but outside of that they were largely unknown.
New Yorkers despise trump and have for ages. From tenants, to contractors, to business partners they all know what a shitheel he is. Outside of NY though trump was largely seen as an affable caricature of a rich dude. Case in point: NYC as a whole isn't averse to republicans and yet they still rejected trump in 2016.
Elon's the same way. Sure he called that diver a pedophile. But he also had a Tesla employee hounded until he fled the country. Musk had the guy swatted for fucks sake (or tried to at least). Or what about the journalist who pissed him off and ended up having some Tesla features disabled on his car. Or his ex-wife who talked about being treated like an employee. These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight. Now the whole world can see that he went full on Howard Hughes.
<< Outside of NY though trump was largely seen as an affable caricature of a rich dude.
This is a very interesting statement to me. I am trying to recall conversations with my wife's side of the family, who supported Trump ( initially, mostly ) and their impression of him was almost exactly that ( plus, "he is a businessman so he knows the game" or something to that effect ). I am in in Chicagoland, but now I am curious if any serious study was done on whether that is how he was perceived as ( ideally before it became socially awkward to state your political affiliation openly ).
<< Their inner circles knew how rotten they both were, but outside of that they were largely unknown.
And that is the other interesting thing. Prior to whatever caused this change of hearts, I only saw impressions of Musk that were similar to impressions of Gates ( you know the ones - saving Africa or something; I know it reads awful, but I really don't remember what his PR presents him as ) - visionary making cameos in odd cultural vignettes ( I think he did one in 'Big Bang Theory' ). The articles describing his work style were very slowly bubbling up.
<< These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight.
And became too much of a power center himself.
I will need to sleep on it, but I think I agree with you.
It's because even though Elon Musk is no right-winger, he voted straight Democrat his entire life until very recently but then he voted for a Republican. The haters will brook no deviation from their accepted political narrative, and when Musk did it the hordes of people with pronouns in their bios turned on him.
For those people it has nothing to do with technology, which is why their takes were so hysterical and wrong. It's 100% about politics every day, 24/7.
Dems didn't want to throw him out. Pence was a competent politician. Trump wasn't. They wanted him embarrassed, not out. Much like the Brits had a plan to assassinate Hitler before D day. Choose not to. They were afraid he would be replaced by someone competent.
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.
I had MySQL running on some bare metal for many years without a restart.
I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully.
Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...
Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter.
But I'm going to assume it's not relevant.
P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of frigging Twitter.
The infra usually matters way more than the code. RAM or a disk will typically fail before the Linux kernel, and it's written in the boogeyman language.
> all software goes down and requires specialist intervention eventually
Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?
If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.
If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.
So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.
Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge.
I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.
However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.
Right, but Twitter wasn't a healthy business (in the sense of being profitable most years) in the first place so it's not beyond the realm of possibility they took reliability further than made sense. Anyway they now have a huge debt load that changes the calculus regardless.
> I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.
Anyone who has ever been oncall can intuit how often stuff breaks in big or little ways. Sometimes it's transient and goes away, sometimes it can be filed away to be fixed in the next year, but sometimes, it turns out to be an all-hands-on-deck crisis for a team, or 5.
> The default, naive assumption should always have been programs keep running indefinitey on their own.
...for people who understand software to some extent. I get the feeling a lot of people see it more like a hamster wheel, where once the developers are gone it immediately starts noticeably slowing down as it stops (and are confused when that doesn't happen).
> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations
Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.
I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now:
1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left.
2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden.
The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.
No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing.
Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company.
The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR money back with a profit, leave the debt to the company and the sucker who buys it, and look for the next victim/target.
Musk likely went the rapacious route because he made a dumb move, and this is the only way he's getting the money he committed to the deal back.
Oh yeah, and when they buy the company, they are in charge of it. So they can pay themselves whatever they want in executive bonuses / etc.
The people who buy the debt know what they are getting into. (And if those 'suckers' don't know, honestly, they shouldn't be investing in junk bonds etc.)
It is not a bad thing as such. The main problem comes when a business is stripped of all its assets and/or weighed down with more debt than it can repay and fails as a consequence, leaving employees out of a job and communities without a formerly productive enterprise.
It is more that it tends to beat raw deal for the purple who lose their jobs for no good reason. The Twitter people will likely mostly land on their feet but most leveraged buyouts aren't affecting highly compensated employees with highly in-demand skills.
Why is that possible or legal? It feels like a loan for purchase of a company necessarily should belong to the purchasers of the company and be paid by the sale of the equity they bought or profit. This way just feels… rigged.
> Yeah, it's private equity 101. I can't believe it happens.
Why not? It's pretty good for all the stakeholders that have a say.
The old shareholders sold for a nice price.
The new ownership didn't have to pay for the whole thing.
The lender gets to charge a pretty good interest rate because there's a good chance of default. If they're lucky, they get repaid; if not, maybe they made enough in interest to make it worthwhile; maybe when it defaults, they'll be able to make something worthwhile out of the wreckage they got at a nicer price.
Leveraged buy outs aren't great for stakeholders that don't have a say. Employees usually get new terms worse than the old ones; in this case, there's been some severance at least. Customers get a promise of a big bang bankruptcy in the near to medium term, rather than a slow fizzle. Sometimes companies with a large payment they can't make can restructure, and sometimes they shutdown with little notice. As a private company without public accounting reports, there will be a lot of guessing about revenue and debt service.
Which goes to demonstrate the very sorry state of our society, a society where employees are not (anymore) part of the "stakeholders that have a say" group.
Because at the end of the day, as you mention, it's the employees and their families that will suffer the most. But as long as those employees don't have board seats while strikes and labor-related physical protest movements have become a thing of the past then I guess this is the reality we'll have to live on for the foreseeable future.
It’s quite bad for the business as a going concern and its ability to serve customers and employ employees when it suddenly, and for no obvious reason, takes on tons of debt. Money isn’t free and loans demand payment.
In Twitter’s case, its $13B in new debt on the balance sheet means that, every year, they have to come up with $13B times the interest rate in additional revenue and/or reduced cost merely to be in the same place profit/loss-wise. Elon already massively overpaid what Twitter’s business-as-usual can generate even before accounting for that $13B; as a result, the post-acquisition Twitter has to try a bunch of negative-expected-value moonshots in the unlikely hope that one of them hits against the odds and the others don’t turn out fatal, because doing nothing or iterating sustainably kills the company via debt service.
In a lot of cases the best solution would be for the company to declare bankrupcy, reorganize, and discharge the debt by convincing creditors to take pennies on the dollar and a share of the resulting smarter-run healthier company, and a judge that the plan is reasonable. However, Elon both poisoned that well by firing people, angering advertisers, and bumbling around product-wise, and also staked a bunch of $TSLA that would need to be liquidated to go through with the bankruptcy.
"Chapter 7 of Title 11 of the United States Code (Bankruptcy Code) governs the process of liquidation under the bankruptcy laws of the United States, in contrast to Chapters 11 and 13, which govern the process of reorganization of a debtor. Chapter 7 is the most common form of bankruptcy in the United States."
Airlines get free government money though. Will twitter? If not, Twitter will need to borrow more money and at higher interest rates compared to market rates if they go through with bankruptcy.
Funny thing about the bankruptcy hypothesis is that the owner of Twitter is the richest man in the world and could literally pay old twitters expenses for decades even if it never made a penny. If we are just talking about the debt payments, he has a 200 year runway.
Actually be probably can't. Musk's wealth is largely on paper, as Tesla stock. Which is enormously overvalued and has been plummeting since he bought Twitter. If Twitter needs cash injections, he has to sell more Tesla, which will depress the price further.
A billion dollars a year of stock sales is a huge amount of guaranteed supply to the market. And it's non-optional: it has to be a billion dollars a year. But Tesla isn't worth anywhere near it's current market capitalization either.
I think that share quantity and value are very separated concepts. At some point Elon loses control of Tesla and that's it: Tesla's price has been tanking, and it's still on its way down to a sane valuation.
3 Renault Nissan Mitsubishi Alliance 7,680,014 (same) -1.3%
4 Hyundai Motor Group 6,667,085 +1 +5.0%
5 Stellantis 6,583,269 +1 +5.2%
6 General Motors 6,291,000 -2 -7.9%
7 Honda 4,121,000 (same) -6.5%
8 Ford Motor Company 3,942,000 (same) -5.9%
9 Suzuki 2,763,000 +1 12.9%
10 BMW 2,521,514 +1 +8.5%
Notice who's not in that list?
Now let's look at market capitalization[2]:
1 TeslaTSLA $568.87 B
2 ToyotaTM $202.86 B
3 PorscheP911.DE $101.44 B
4 BYD002594.SZ $86.39 B
5 VolkswagenVOW3.DE $84.30 B
6 Mercedes-BenzMBG.DE $69.28 B
7 General MotorsGM $57.48 B
8 BMWBMW.DE $57.39 B
9 FordF $56.60 B
10 StellantisSTLA $48.87 B
There is no possible way Tesla is worth more then double what Toyota is, while shipping about 1/10th of the vehicles. The Price-Earnings ratio is 56, vs Toyota's 9.
By every possible metric, it is vastly overvalued compared to it's profitability, and there's no reason on the horizon to think it will suddenly catch up since all those automakers are moving directly into it's space.
It's borderline a meme stock at this point. Tesla is valued as if it will forever be able to sell every car they make, on earnings calls Elon blows off any questions about demand. In the last few months though used car prices for Tesla have plummeted, I think they're finally about to run into demand issues as they get more supply online
If you want to compare companies by 'how big' they should be, you need to compare enterprise value, not market capitalisation.
If you think Toyota's stock (or bonds etc) will increase in market price relative to Tesla, for a fee financial markets will let you bet on that conviction and make money, if you are right.
I also want to add a #3: the crew that's left is probably on-call 24/7. My thoughts are with the poor souls on that rotation (if your team even still has a "rotation")
To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.
Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!
To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.
As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.
Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.
And the house didn't fall down within 3 weeks of it being purchased.
Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.
So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.
Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture).
Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement.
You may not like it, but having a bot do that probably saves a lot of legal hassle.
> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.
If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.
Hm? Intellectual property is explicitly carved out of 230, and even if it wasn’t: it isn’t user generated. Content providers are regularly found liable for infringement on their platforms, especially when the plaintiff can demonstrate willful negligence (which in this case would include discontinuing a seemingly effective scanning system.)
Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.
Well, Disney won’t care why their copyrighted material is publicly available, noone likes this sort of copyright, but if Elon wants to avoid huge fines he better (make someone) fix it ASAP.
Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.
It’s more than that. For one it’s not an existing boolean, there are now two different kinds of blue tick that are presumably stored separately. Blue is also supposed to give the user fewer ads (while making them more relevant) as well as additional weight in feed ranking algorithms. It’s also intended to be offered worldwide which adds a lot of complication to things like payment flows.
I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.
Clearly worldwide payments is an issue otherwise they’d have rolled it out worldwide day one, and they didn’t. There must be something holding that back.
Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.
In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?
He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.
As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.
Well, define 'failure'. Minor outages like the one you are talking about were happening from time to time long before Musk bought Twitter, and it even suffered long outages frequently - remember all those fail whales?
I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.
failure. noun. with no security team, hackers are able to get in easily.
everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.
Yeah, I don't think anything remotely close to it will happen, unless some of the fired developers have left themselves some backdoors which they'll give (sell) access to it to someone.
does this stuff happen automatically? is there a robot that goes out and reads about all the new zero-day exploits and patches all the software without human intervention?
On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?
Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.
Pretty sure that the developers that are left will not be liable for anything unless they are knowingly participating in criminal activity such as criminal negligence that is the direct cause of someone getting seriously injured or killed.
Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.
For now I'm talking about the FTC consent decree, so administrative penalties not criminal charges. Musky fired the folks who were responsible for ensuring compliance.
How do you propose Musky does that with no payroll department? E-Lon walked back the mandatory return-to-office policy last week. Surely if he could find (or thought he could find) suitable replacements he would be pretty comfortable demanding RTO.
I'm not really familiar with how they do payrolls in USA, but I'm pretty sure it isn't some rocket science (pun intended) and can be done rather cheaply by an outsourced firm.
Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?
It's simply an abbreviated version of Elongated Muskrat as far as I'm concerned. Payroll is easily one of (if not the) most complex systems at any company. It's not just statutory stuff but personnel stuff as well. With Twitter you're not just dealing with 50 states and the feds, but with every other company in which Twitter has (had?) employees. There's a cottage industry of payroll firms precisely because payroll is so obscenely complex.
Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.
If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.
Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.
> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.
Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?
> but it was never going to happen overnight and I'm not sure why news outlets were saying that
I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.
This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me.
Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it.
People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems
>I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue
In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done.
Not quite the same business, but a bit reminiscent.
I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.
true. if twitter, Facebook, reddit, and hackernews go down for a couple days it wouldn't affect me at all. if GitHub and npm went down I'd me mildly annoyed but could still work.
I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.
Undoubtedly some people will be put off, but think of how often Reddit used to go down -- still got pretty big. And Twitter already has all the newsmakers people want to see. If your goal isn't necessarily user growth it makes sense
I have personal experience with this. The metrics (as much as I despise using them as a source of truth) undoubtedly show a very strong positive correlation between better load times and user retention.
We had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).
I've never heard of (or thought of) your interpretation of the corollary to Brooke's Law, but removing people from projects until they succeed and are on time seems like a bold strategy.
Happens regularly in the Free Software & Open Source world, where software projects are often forked by a single individual. Obviously enterprise software is on another level, but the principle remains that reduction to a smaller development team (for some period of time) does not necessarily correlate to a threat to viability and has often indeed been a reinjection of vitality.
Homeostasis is a good metaphor, but it implies a living, dynamic system. Something that resists entropy by itself being in a state of flow -- the matter constantly changing, while maintaining the form.
In modern software environments, the entropy is almost violent -- the changes in all the constituent dependencies are constant and relentless. Something frozen in time does not stand a chance, unless it's entirely stand-alone and dependency-free -- an unlikely scenario with a service of Twitter's size.
IBM is a "tech" company that employs 282,000 employees, and when was the last time they invented something? I don't remember the last time I heard IBM in the news about something they made.
The bigger the company, you often times find less innovation and more administration & bureaucracy.
The reason startups can survive is because of its small size that makes it very flexible and adaptable to chaos and change, that gives it the edge over bigger companies.
All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics.
Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart.
The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or elicits help from others on what to say that sounds like tech speak (RPCs!), but when it comes to being truly knowledgeable in this space, he is losing his most valuable assets because of his amazingly poor managerial and ownership style.
I know there are a lot of Elon fans on this site, and will disagree with all of this; but his abilities have not at all been proven. Yes, he knows how to spend money to claim credit for technical advances, but until he actually has his hands dirty in the muck of the hard work of tech, he will always be a glorified self-promoter with no substance.
And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!
I say everyone can join for free and gets one post a day. $8/month plan includes 10 posts a day, and unlimited for $20. While I have no interest in Twitter, I'm told it's quite addictive, so make the first taste free and then reel them in once they're hooked.
What choice do they have? The company was hovering around profitable (2021 would’ve been without a lawsuit settlement) but that was before they were saddled with a ton of new debt. It’s possible that they could find new revenue models - that pay for checkmark scheme isn’t it but maybe a smart businessman could come up with a better variant - but that takes time and has to be done carefully since they have to keep the lights on in the middle. Driving away advertisers before finding that new revenue source doesn’t leave much time to iterate.
Debt and equity are just different ways for investors to invest in a company. If company isn’t generating any return on investment, keeping the lights on is not sustainable.
If that's it's business model then sure, that's how it works.
A "de facto public square" would be public in conception, construction, and support from the start, which is one of the ways we know that Twitter is no such thing. Though it would likely also have some rules for how speech is/isn't conducted.
And all things considered, advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse.
> advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse
This reminds me that progressives have historically always supported corporations as complex hierarchies, scientific enterprises, run (ideally) by “experts.”
Nowhere was a claim forwarded it was progressive or ideal. The claim is more or less that advertisers have some of the same concerns that elected representatives do because consumers have something like a vote.
But then again, if you're scare-quoting expertise, maybe that's not the conversation you're here to have.
I'm scare-quoting expertise because it seems like you think there's an expert way to run a corporation.
And that's a very progressive sensibility towards corporations, and precisely why they actually like them (what's more appealing to a progressive other than a huge centrally-planned organization run by credentialed experts) despite claiming otherwise.
LOL. If you think that progressives are generally friendly towards corporations then you've never actually talked with a progressive, have you?
And corporations are not generally progressive in outlook either, there are too many values higher up on ladder of concerns.
Advertisers are one subset that bring a rough approximation of democratic to their decisions, knowing that each person in the market they hope to reach will be deploying something like a vote with their dollars. As with any democratic approximation, it's only progressive to the extent that population is, though it's poor compared to other democratic mechanisms.
You can take issue with whether democratic decisions are good decisions, of course, but that's likely to be an unpopular opinion for obvious reasons. And hey, I hear the person currently running twitter recently went so far as to say vox populi vox dei. Was his expertise in running corporations part of what you meant to scare quote, or is your challenge to expertise, shall we say, selective?
Read up on Progressive history and, yes, it's very clear that the early 20th century Progressive movement loved the concept of a corporation.
While most aren't self-conscious enough to realize it now, they still love corporations as evidenced by how they zealously defend them when they serve progressive ambitions and power.
That depends on Musk’s goals. If his goal is to make money to pay off the massive amount of debt he gained as part of buying twitter, (as the attempted quick roll out of twitter blue would indicate), then yes he probably needs to care about what advertisers think. If he just wanted to leverage his position as one of the richest men in the world to ensure that twitter was a haven for free speech and screw the profitability, then no he wouldn’t need to be guided by advertisers. He probably can’t both want immediate returns on his investment and to quickly rock the boat though.
>while advertisers are fleeing because Trump is back on
[citation needed]
CNN's ratings were never better than under Trump. He's fantastic for advertising. So is Musk. All controversial figures are. That, oddly, isn't controversial in advertising.
>on a whim!
He created a public poll, and when people voted for Trump to be allowed back won, he unbanned him, tweeting "vox populi, vox dei" ("the will of the people is the will of god"). Had he unbanned him despite the poll saying "no" you could argue it was a whim, but that isn't the reality we're in. He also refused to unban Alex Jones, citing exploitation of child deaths and a personal story. Not unbanning Alex Jones was more whimsical than unbanning Trump was, factually speaking. Why do people always misrepresent his actions? And why is it always upvoted and not flagged here?
These people behave just like the irrational fanboys, except they just do the exact opposite. Being a sheep and being a contrarian sheep are the same thing.
Honestly, I'm hopeful that other coastal tech companies do clean house. I genuinely believe it could lead to a resurgence in the lives of Midwest/middle American states and cities.
There is an absolute vacuum of technology specialists in the middle of the US, because no one wants to "live in the middle of nowhere," and they don't want to earn less than FAANG (MAMAA?) salaries, when half of those salaries can give you an amazing life in the middle of the country (source: my piss-poor salary compared to yours).
You aren't wrong, but his playbook is familiar to anyone who's gone through acquisitions (especially leveraged ones) and many companies were in a strong enough position to start with that they do manage to limp through and get sold off despite all the abuse.
Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but that's hardly required in his situation.
America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.
Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not.
It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increasingly erratic behavior in the meantime.
Is today's Elon capable of doing what Elon from 15 years ago did at Tesla? I don't think that is necessarily in evidence, much less in a very different industry.
What I think makes people skeptical of him is that deep down we all believe in nominative determinism. Who do you think runs a rocket company behind the scenes, "Shotwell" or "Musk"?
There's a lot to unpack here, for one do we consider those two enterprises to be successes? It seems too early to tell - they do appear to be influential, but the C64 and Palm Pilot were influential without being successful. It's also not clear if they are long term successful (I think Tesla will be, but SpaceX is very much currently dependent on government funding). Finally, it's not clear whether Musk was a critical driver of success - evaluating his contributions based on the outcome of a company is basically resulting.
Look, I don't know if Elon is a genius or an opportunistic parasite with really good PR. It seems unlikely if we ever will know that. What I object to is people pointing at his ultimate financial success and crediting him with the current result of 2 big companies whose future is very much not determined.
When I look at his process from this ant's perspective, I think he is an abusive unstable individual who takes credit for everyone's work and lies a lot. He also flip-flops depending on the wind. Is that success? Not based on my personal values. Have his companies accomplished a lot? Some of them, absolutely.
The definitions and evidence matter a lot, and I personally don't think any of us are qualified to make blanket statements based on incomplete outcomes. Further, I don't think his other companies that require primarily good engineering are very relevant to inherently people problems, like Twitter. My evaluation of how Musk handles people problems is that he is very bad at them, and I anchor my prediction about his Twitter leadership based on that.
> Automotive and aerospace are not that similar to social media.
Yes. Social media is easier.
> It's the Halo Effect fallacy to think competence in one field automatically translates to another.
This is precisely about leveraging the Halo Effect fallacy. Elon Musk might not know social media, but the markets don't know that, nor do they care. The average retail trader sees "Elon Musk's company" and buys and holds, regardless of absurd PEs.
Musk knows the power his brand has. He's simply going to use that to pump up Twitter's valuation, all through the virtue of his "halo"
Dealing with people is generally far harder than pretty much any engineering problem. The same is true of Twitter, because there are no easy answers or even clear goals.
The easiest way to do this would be to make the site completely free of moderation, but that quickly becomes a cesspool. Are there other ways to define the core audience?
Exactly! I can't fathom that people don't seem to understand there's a comparable amount of new companies started every year in the space, automotive, and social network categories.
This is just a misdirection. I could just as easily said, nobody did it by themselves, including Musk.
What is “it” in this case?
NASA (and the DoD) had vertically landing reusable rockets designed for orbital flights back in the early 1990s. They were being successfully tested but budget cuts killed the program. They weren’t doing “it” because it wasn’t the same priority in that era. NASA has been researching COPVs for decades, etc.
The SpaceAct agreement between NASA and SpaceX allows for sharing of this kind of information. If you think SpaceX has done all their great work alone, you are misinformed and likely making you data fit your conclusion instead of the other way around.
SpaceX has some competitive advantages, but I don’t think they are what you think they are.
> If you think SpaceX has done all their great work alone
What I'm saying is SpaceX got it done. No other organization in the world did. The fact that SpaceX had an obvious learning curve of failing and exploding rockets makes it obvious it wasn't just copy and install NASA technology.
NASA's reusable rockets were on the space shuttle, which turned out to be fantastically expensive and impractical.
> NASA has been researching COPVs for decades
Somehow not resulting in a practical, inexpensive reusable rocket.
It's undeniable that NASA has made many great achievements. But making space accessible in an economic manner isn't one of them.
>NASA's reusable rockets were on the space shuttle
I wasn't referring to the shuttle. Note I said "vertically landing reuseable rockets".
Are you claiming SpaceX doesn't use COPV's, or that don't benefit from prior COPV research? I don't think either position is accurate. Of course no single technological advancement defines space exploration.
Can you elaborate on what you think SpaceX's key advantages are? I can use that to gauge if you really know what you're talking about. I don't want to sound rude, but it's starting to come across as a poorly informed discussion, but one where you have strongly held beliefs. That's not the relationship we should probably hope for.
I already said the NASA/DoD program was scrapped in the 90s. They had successful sub-orbital test flights, the original designs were for an orbital craft, but the project was canceled before that could be tested.
I meant what advantages do they have to facilitate that. You gave me the outcome but haven’t shown any understanding of the why. When somebody asks what makes Tom Brady special, saying “because he wins more” isn’t really saying much and doesn’t take show you know much about football.
I ask because I suspect you will just give some rote public vs. private answer but that’s only a superficial reason. There are underlying systemic reasons, but you need to remove yourself of that false dichotomy first to get there.
FWIW, I’m not a big fan of NASA. I think they are largely a broken culture and a shell of what they were in the 1960s.
This is exactly the kind of vague, hand-wavy answer that I was hoping wouldn't be the response. It doesn't really show any understanding of the problem. The issue with the "profit motive" as an explanation is that it's so vague it can be used to argue both sides at the same time. The profit motive helps them pay for the best-and-brightest. It also biases them to hiring the least experienced at a cut rate. It incentivizes them to provide the best product. It also incentivizes them to cut corners to save money. It explains why SpaceX has been able to move fast; it also explains why moving fast caused such problems with Boeing's Starliner. So the "profit motive" doesn't really explain anything. Besides, the vast majority of NASA work is done by for-profit contractors and has been since the Apollo era. There's nothing new about it in spaceflight.
I'll try to illustrate a more nuanced perspective. It's no secret that NASA levies a lot of tough requirements. For example, contractors must have a robust pressure systems program. This includes managing/certifying systems all the way down do small air compressors in a vehicle maintenance shop. Same goes for software quality and a million other aspects of spaceflight. Contractors hate these types of requirements because they're expensive. Many within NASA hate them, too. Certainly some of this is bureaucratic overreach, but a lot of it is also good, sound engineering practice. There are mechanisms to waive these requirements, but few people want to openly do so for a variety of reasons. I could go deeper into the why but it's a bit of a digression.
So what does this have to do with SpaceX? CCP, IMO, is a clever work-around to avoid accountability to these requirements. NASA, rather than buying a product, is buying a service. So even though NASA expects them to meet those same requirements, there's very little oversight to force them to do so. Some people have raised flags about these issues but are essentially told to stand down because they don't want to tell the contractors how to provide the service. It also gives NASA a smokescreen to get what they want (faster, cheaper production) while avoiding accountability when things go wrong (they can always point to the requirements they claimed they wanted, but didn't provide the oversight to ensure). NASA knows those requirements rapidly increase costs and on one hand they don't want them, but on the other they want plausible deniability if something goes wrong. Minimizing requirements can streamline the process. The fact that they manage way less requirements is why SpaceX can have a single 23 year-old managing a program that takes a team at NASA. In other respects, it turns a blind eye to the very requirements that manage risk.
E.g., Falcon 9 had supplier quality issues that lost a rocket [1]. Most who work with flight hardware would be surprised to learn SpaceX wasn't applying industry-standard supplier quality checks on critical flight material. Once a mishap happens, NASA gets to swoop in and investigate. And the result is SpaceX now has multiple reliability layers to mitigate that risk. It's not that it was some unknown risk, it's just that they weren't managing it properly. To the uninitiated it looks like they were running a tight, streamlined ship but in practice it was being played a little too fast and loose. Boeing did the same with Starliner but had a bad roll of the dice. The real question is how many times can this happen before SpaceX starts to look like their bloated competitors? NASA could do the same by just peeling back requirements and upping the risk.
Starliner also had it's own host of quality issues as part of CCP. People at NASA were concerned, but their hands were essentially tied until there was a smoking gun in the form of a botched demo that risked crashing into ISS. Again, the "profit motive" at work can sometimes mean more risk than intended.
John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so."
Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years."
Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being technical in interviews is all set up and that he was trained on the materials and questions ahead of time.
With a handful of tricks or a patsy in your pocket, it's easy enough to pull things like this off. These are all self-reported encounters, which lends some doubt to them; as I've never seen any public performance of his that suggests he has this exceptional intelligence or that he isn't subject to the same amount of irrational thinking that most humans are. You may be able to do some type of rocket equation in your head, but if you constantly promise things that aren't ultimately delivered.. people have good reason to question this narrative.
He clearly does know how to make incredible sums of money. Why that's not enough and people need to find excuses to exaggerate or demean his intelligence is beyond me.
Maybe I'm a particularly dull engineer, but I've taken several aspects of personal advice from what he has said in interviews (the especially technical ones, not the ones aimed at a mass audience where he repeats his standard canned speech) and found them useful for myself personally.
Here's two examples I've found particularly insightful that shows he has some ability to talk about engineering details.
This example where he talks about the choice of steel for Starship as opposed to any other metal, something that would be an otherwise unsual choice: https://youtu.be/vLC5W53Fsyg?t=936
This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805
>This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805
I knew what you were talking about when I started reading your comment. "Make your requirements less dumb" first seems so obvious once you've learned it.
All the denigration directed at him seems to come from people who've only read headlines about him from sources who hate him.
> but if you constantly promise things that aren't ultimately delivered.. people have good reason to question this narrative.
This is the insane thing to me. He's promised a lot of things, but he has also delivered some pretty huge things. Tesla kicked off the electric car migration and has millions of EVs on the road. SpaceX has reusable first stages on their rockets and are the only private company to send humans to space. Just those two things alone are massive achievements. But people look at some things he's promised but has not yet delivered and that somehow is more important than what he has delivered?
I'm inclined to believe this is true. The problem is that Twitter's challenges are social/political, not technical, and Musk has demonstrated little competence in this area.
This is the absolute root of what's going on, right here. Twitter is only nominally a tech company; it's a media company. It may be that he had to cut the fat over there, I don't take issue with that necessarily (though I certainly do take issue with the disrespectful way he went about it), but image is incredibly important at a media company and he's notoriously bad at comms except with a small subset of people. Twitter needs sensible policy and thoughtful communication, and he wants to ram his ideology through it like he would shake up any technical process.
Another area where Musk's "brilliance" has faltered is in his transportation ideas and there's a connection here to twitter in that the challenges here again are not technical, but rather political around land use.
Brute forcing a problem with better technology not always the actual solution when technical problems aren't actually the problem.
Something doesn't compute in this scenario though. Either his tricking everyone around him or is unfortunate enough to slip up publically. Not knowing what GraphQL is and talking about RPCs in HTTP is a very revealing slip up.
My guess would be that he has some knowledge but also is very good at faking it which is not necessarily a bad thing - those are good traits for a CEO. Though people should be aware of this fact when evaluating the whole persona.
> "Anyone who actually writes software, please report to the 10th floor at 2 pm today. Before doing so, please email a bullet point summary of what your code commands have achieved in the past ~6 months, along with up to 10 screenshots of the most salient lines of code"
Actual quote.
Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is out of a Dilbert strip.
"Code commands" is very plausibly an autocomplete flub of what was supposed to be "code commits." When I type "code comm" my iPhone offers up "commands" as the completion.
I've seen a lot of mockery of this request, but I suspect people aren't considering the wide variance in employee quality that can exist within a mismanaged organization. What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.
> What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.
So it's basically a FizzBuzz test, but for existing employees?
Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it works about as quickly as you'd expect a PhD student to. The fact that he can do this across many technical disciplines is impressive and borderline superhuman. I don't think people understand or would believe how low-level and technical typical meetings with him are. Just saying because I get triggered reading way off innacurate takes on this topic (original comment).
If that is the case, why has he been making mistakes that seem fairly elementary on Twitter? Like, I understand not understanding a problem space and wanting to learn more. But you say he has good intuitions and mental models–I would've expected at least some basic background research before posting online. Why aren't we seeing that?
I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable.
You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It."
That's what's happening here.
That engineer that was humiliated publicly for defending a slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine? Hacker News is filled with people just like him that have built near identical software in other orgs.
Understandably they're upset and are looking for any excuse to dismiss Elon's criticism of not just Twitter, but their entire industry.
I doubt you'll find many people arguing that the value of employee output is evenly distributed across all people in a company: regardless of discipline, there's always a minority who are delivering the most value. There's always the most valuable software engineers, the most valuable sales people, the most valuable executives. The problem with Elon's specific brand of this take is that it's ignorant of the real-world human aspect: Elon could pick the best software engineer he has ever worked with, and helicopter them into a dysfunctional environment, and they would struggle to deliver value.
If Elon had joined Twitter, and spent time understanding the business and environment and then excised the people he felt weren't contributing towards his vision, that would be one thing... but he has made arbitrary judgements based on absurd metrics like lines of code or willingness to show up at 1am to draw on a whiteboard, he has not made judgements based on the quality of the work or the value people have delivered.
Likewise, to suggest that a software engineer is bad because they were a part of a team that built a "...slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine..." is absurd: what if that person was the only reason that it wasn't 10x slower? What if, they were the lynchpin in that team ensuring that brought everyone else up to a much higher standard which ensured that what they built was usable (even if it was bad)? You cannot judge the contribution of an individual without considering the wider context.
I have no problem with a company cutting most of their software engineers (I encourage clients to minimise their exposure to software engineers, I encourage careful hiring over volume) but what Elon is doing is... not that.
> ...spent time understanding the business and environment...
You're probably right, but there's a decent chance that Elon's heavy-handed approach was necessary.
I've seen the "gentle" approach fail.
For example, at $dayjob a bunch of on-prem stuff is being slowly modernised into the cloud. Very slowly. Slow enough to give the dinosaurs time to play politics and protect their turf.
For example, the networks teams that are used to legacy in-line firewalls will cozy up to some non-technical senior manager with a budget and get them to approve a project to roll out this legacy technology in the cloud. That way they don't have to retrain or -- worse -- risk being made redundant.
If instead some team comes in and simply bulk-migrates workloads from on-prem to the cloud... breaking a handful in the process and just fixing forward, then it appears to be messy and crazy, but the effect is that the legacy data centre teams are made redundant virtually overnight. Now they've got no clout, no time, and no pull. They're simply walked, and will find jobs elsewhere.
I've seen both approaches, and the latter style worked better long-term.
Every org has slow-as-molasses, badly designed, illogical components. It is about ratio.
Time will tell. I don't think Musk/twitter's case will set any precedent. He is too much of a character to provide broad meaningful insights into industry. Also he has accumulated a list of failures which are rarely mentioned.
Has he ever make a public statement, at least a paragraph in length, explaining something technical?
It's hard to blame people based on his decades of public behavior and lying about his education, falsely claiming to have a physics degree and to have been admitted to grad school.
Adding another datapoint from one of my previous comments:
In response to someone saying on Twitter how Elon doesn't understand the technical stuff of rocketry, Tom Meuller, former CTO of Propulsion at SpaceX and the designer of many of their engines responded
"I worked for Elon directly for 18 1/2 years, and I can assure you, you are wrong"
Channing Robertson, the face of Stanford chemical engineering department and the associate dean of Stanford’s School of Engineering, who taught and mentored Elizabeth Holmes, has said the following to say about her:
“She had somehow been able to take and synthesize these pieces of science and engineering and technology in ways that I had never thought of.”
“I never encountered a student like this before of the then thousands of students that I had talked”
“You start to realize you are looking in the eyes of another Bill Gates, or Steve Jobs.”
He also maintained that Holmes was a once-in-a-generation genius, comparing her to Newton, Einstein, Mozart, and Leonardo da Vinci.
Excerpt from: "Bad Blood: Secrets and Lies in a Silicon Valley Startup" by John Carreyrou.
You know, I think Musk is an ass, and would never work for him, but don't you think that someone who has managed to launch and then run many successful and complex technology projects might actually know a thing or two about launching and running simpler technology projects?
And if you're going to claim that his successes have been due to the people surrounding him who actually know what they are doing, then all that tells me is that you are acknowledging that he knows how to surround himself with people who know what they are doing.
We're not fans (I'm certainly not), but it takes a special kind of mind to look at Musk's track record of successes and conclude that his latest project is doomed.
Well, I think the issue is precisely considering Twitter a "simple technology project", and it's the same mistake that Musk does. Twitter isn't a "software and servers business" as he said. Twitter is a social community, and while in some regards it might be easier, it's also far more difficult in others. Just compare how many business and institutions can reliably launch rockets or create cars, and how many can reliably create social networks.
Point still stands if you call it a "simpler technology project". It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.
> Point still stands if you call it a "simpler technology project".
Which point?
> It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.
While I broadly agree with you, and also agree that twitters founders and/or leaders are better equipped to manage a community than Musk is, I fail to see how having someone with a track record of success (some in turning around failing businesses) automatically dooms the company.
That was the point I was responding to, in the original post. It's also why I ended of with "it takes a special kind of mind to automatically assume something is doomed just because someone with a track record of successes took it over".
I don't think Twitter is automatically doomed, but I think there are more things to consider than just Musk's "track record of successes". And one of the important things is that he doesn't look like he actually understands Twitter. He keeps calling it a "software and servers company", keeps talking about "hardcore coding", alienating and angering users and advertisers... People would be be more confident in his abilities if he actually looked humble enough to recognize the parts where his expertise is lacking.
I think it's somewhat reductionist to call Twitter "simpler". The technical challenges faced by SpaceX, for example, are almost completely orthogonal to those faced by Twitter. Imagine swapping a random engineer at SpaceX with a random software engineer at Twitter -- do you think either would thrive in their new role?
Compare how many companies can reliably launch rockets vs how many can reliably create social networks that aren’t dead in a year. I think it’s clear what’s more difficult
On the contrary, I'm pointing out that "simpler" is meaningless in this context. The two companies face fundamentally different challenges and require completely separate skill sets. Some people will be better suited for Twitter, whereas some people will be better suited for SpaceX. For the same reason that you shouldn't commission a mathematician to remodel your bathroom, you shouldn't expect the CEO of SpaceX to successfully manage a social media company.
For further reading, I'd recommend this article in the Atlantic. The relevant portion is about midway through.
You underestimate Elon Musk. Many people have done that before and lost that bet. If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies. The notion that he won't be able to attract world class talent is ludicrous. Yes, he is a bit of a liability and his management style is obnoxious and unconventional. But he does get things right once in a while.
And he hates bloated inefficient teams. His decrees on meetings are infamous. Tripling the team at Twitter implies a lot of internal politics, fiefdoms, communication overhead, and generally a lot of headless chickens running around. There's no nice way to fix such a team. A sledge hammer is one way to fix it and obviously he likes getting results quickly.
So, the notion of laying off most of that team was a foregone conclusion. The notion that a lot of the better people would get upset about that and leave as well is also highly predictable. What's left is a team with some gaps but also a lot of breathing room. And he can always lure key people back in by throwing money at them.
Simple plan. It might actually work. At the cost of a bit of drama, temporary instability, and lots of free publicity. Exactly his style. Cringe worthy and effective. I can see the logic here.
Agree, I wonder how much it is thought through strategy and how much is just "natural" style applied indiscriminately. I think one more important part is that he has money/resources to be able to make mistakes without bankrupting and stubbornness to plough thru even when things go wrong.
> If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies
The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly manage the community and make the company profitable.
Twitter doesn't have a tech problem, it has a community problem.
Exactly. it had a team problem. I think it's safe to use past tense now because that team is mostly gone now. It still has some team challenges but those he can fix with strategic hires and hard work.
Fixing the community starts with rolling back all the things that clearly did not work. He's using the sledge hammer method there too. So, not very subtle but generally just getting of rid of a lot of failed and failing policy.
The technical challenges in speed and scaling are not challenges at all anymore. Twitter built a lot of stuff in house when you couldn't get that stuff as a commodity. That has changed since then. You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
> Fixing the community starts with rolling back all the things that clearly did not work.
And which ones are those? Knowing what did and did not work is an actual challenge by itself.
> You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
The massive technical challenge is migrating existing infrastructure to something off the shelf, then finding and fixing the new bugs in that existing infrastructure and/or your deployment/configuration. That shouldn't be underestimated.
> The massive technical challenge is migrating existing infrastructure to something off the shelf
It's short term disruptive and might involve some fail whales. And then it is solved again. Break it, fix it. Like it or not, that seems to be the plan. If you accept things might temporarily break, it's going to be a lot easier to act.
It's a microblog, not star ship. At least from Elon Musk's point of view, there's going to be a difference between those two and the amount of brain cycles he's going to dedicate to things breaking or not. He's not going to be afraid to break the team (check), the platform (still running fine), or the community (in progress, engagement seems up so far).
Years ago I had a massively downvoted comment when I criticised his AR for CAD vapour ware. As someone who was fully in that area at the time, what he was showing while looking fancy had no practical application in the area of design he was talking about.
Ever watch someone do CAD/CAM modelling? They need extreme precision of input that AR sausage fingers just aren't going to help with. You need a num-pad and a good mouse with a stepped click wheel.
I get it. We share some similar neurodivergence traits. He wants to be right in the detail. Constantly jumping from interest to interest, seeing the hidden patterns and connections that aren't apparent to others. But there are a times when I know I just need to shut up and let someone more experienced talk despite my brain wanting to lead every discussion right into solution mode, or providing additional context mode.
I've spent the last 6 years in management consulting (without formal business education), I agree with him when he says MBAs are useless. We know that the best solutions come from diverse teams with diverse backgrounds, skills and knowledge. Not 5 clones who know how to build value driver trees, not to say the tools they bring aren't useful, but they can be incredibly limiting.
For someone who hates MBAs he's sure going about this take-over like someone who barely passed one (i.e. knows more than enough to be dangerous). Sure, you're hemorrhaging money in operations. You need to cut costs and find new revenue streams.
What are your biggest costs?
Labor. Slash / Burn. The old McKinsey 7% FTE reduction will give you some extra operating cash from the years remaining budget and you know it's not so much that people (in fear of their jobs) won't just pick up the slack to keep everything moving. Do it quick because you need to rip the band-aid off and get rid of all that accrued leave, restricted cash etc. off your books too.
Equipment. Redundancy? Sounds like unused resources we can fire sale.
Contracts. Renegotiate? The only two meaningful levers are price and quantity. Start cutting quantity now, renegotiate price later.
This is all dummies guide stuff and tends to go terribly in reality when implemented all at once all together.
For instance, research has shown companies that lay-off when under pressure end up underperforming against the ones who chose not to.
Now who's going to help build and operate those new revenue streams?
Quick fixes for a quick buck and a whole lot of extra risk.
Breach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.
Nothing here seems to approach anything like truly proprietary knowledge and the kind of information it contains is pretty similar to what goes up on engineering blogs (or what you'd describe in a job interview, if you prefer) all the time.
Yes exactly this. Maybe they did but I didn't find it stated in the writeup, which was actually pretty good. If it were my company though, I'd want the intellectual capital protected and we programmers usually have to sign non-compete agreements so I think my original comment stands, it was my opinion and initial gut reaction, not an indictment or judgement in any way.
Who's going to compete with Twitter? It's a company worth billions, and to execute the caches and automatic fallover in the rival company with billions of users it would take billions.
To be hair Musk posted much more internal information in a screenshot lately. The question isn't really is it incredibly useful, but is there anything that might be a breach of the employment agreements. To this I don't know, but it is still an area where employees should be careful. I personally would never write anything about my employer on a personal blog.
Out of all the comments here, I can't believe this is the only one that shares my sentiment. I'm not a software engineer, but in my area of engineering if I ever revealed anything publicly as detailed as this kind of architecture; for instance, the failure modes effect analysis framework use at a certain aerospace company... or details of the simulation models involving the crash analysis at a certain car company, so soon after departing one of those companies, I'd expect a letter from a lawyer within a week. In most places I've worked I don't think I could even post anything on the internet invoking the name of the company unless it was a repost of some public information the company itself had previously released.
Do the vast majority of people here on HN/in Silicon Valley not have 5-year NDA's? Or do they and they just not care?
I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in any way to Twitter.
A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.
The unemployment rate in Feb 2020 was 3.5%; by April 2020 it was up to 14%. It has never been lower than 3.5% since then. If there was a hiring glut due to covid it must have been a very industry-specific glut.
It definitely was industry specific. The need to satisfy a boom in remote working and people bored out of their minds in quarantine drove a ton of hiring in tech. If you had money to invest, tech stocks and companies were among the few places it made sense to pour money into during the pandemic.
There have been several front-page posts on HN in the past few weeks about big name company CEOs apologizing for massively over hiring during the pandemic not realizing that it wouldn't pay off once the lockdowns ended and people decided to go back to living their normal lives again.
What a wild card. Has Musk said why he purchased twitter in the first place? IRL he seems so introverted. It's just odd to me he'd have any interest in a social media company.
He has a unique perspective as someone who's addicted to tweeting and whose account is constantly getting impersonation and crypto bots in the replies. His friends are incompetent VCs and other tech people who are convinced journalists and women with colored hair have taken over Twitter and are hoarding all the blue checks for themselves.
He's also extremely divorced, keeps having off the record children with his executives, and is having a midlife crisis.
Probably a combination of a career-first mentality where they don't have the time or inclination to seek / maintain lasting relationships, and also having the means to not raise their own children. To these people children are not burden in the same way as traditional parents because they hardly contribute to raising them; instead they are just another play-thing.
There may also be some game of thrones shit going on where if a boss and a mini-boss have a baby it solidifies rank and power.
My pet theory is he got roped into it because the SEC was going to hit him for securities fraud (wouldn't have been the first time he'd done it[1]). Earlier this year, he had originally said he was buying a 5% share and would be a passive investor. Later on, whoops it turned out he'd bought over 9%[2] of the company without the mandatory disclosures, potentially defrauding the market of over 100 million that he would've paid in inflated share price if his moves had been known to other investors. With the SEC on his trail he said screw it, and inked a very difficult to escape contract to purchase the company. The SEC gets off his back, and he spends months trying to wriggle out of it but finally is compelled by a dead-end court case to fulfill his obligation.
Musk is an excellent strategist. When he says something in public it's because he wants people to react to what he said. It doesn't mean it's true. I'm not suggesting he 'lies' exactly, more than he uses his audience to manipulate the world around him to get him closer to his goals. His Twitter account is more PR than reality.
I have no doubt he had excellent reasons to buy Twitter, but what he says they are in public probably isn't the whole story.
Can you trust anything Musk says? We know he’s full of shit; ask Thiel. As for the real reason, pick your poison:
- He purchased it because he’s in a right-wing filter bubble and bought into the idea that Twitter is controlled by a secret left-wing cabal and genuinely believes he’s doing the world a favor because he thinks right wing propaganda is actually moderate
- He has the same brain damage as Trump and requires adulation by sycophants and he thinks this is the best way to guarantee his fix and give him control over his critics
- He wanted the brand name and doesn’t care about what services are currently offered under it
- He’s from the southern hemisphere and wants to destroy the social fabric of the United States
- Some permutation of all of the above
Not sure which of these are the truth, but I’m guessing a combination of the first two.
Twitter was poorly managed. It would have been a $100B company in 2020-21 if it had shown any semblance of consistent profitability.
The opportunity here is to:
- Take the company private
- Cut costs by pruning staff and reducing bloat
- Get high-engagement accounts back, which, in turn, will bring back advertisers (perhaps sweeten the deal by promising something with your other ventures)
- Eke out a few quarters of consistent profits
- Go public again, this time with $500M/quarter of profits
- Enjoy the standard "Musk-effect" 40x PE multiple and slow keep cashing out
Hard to cut costs when you've just saddled the company with billions of dollars in new debt. Don't forget El Musk-o is getting sued by Tesla shareholders over his bloated compensation.
Get high-engagement accounts back
Advertisers didn't leave en masse because "high-engagement accounts" left, they left because the "high-engagement accounts" came back. From the looks of it "absolute" free speech doesn't include Alex Jones, so who knows what other "high engagement" accounts will be left out in the cold. Doubling down on "absolute" free speech just means you're going to have a sea of ads about pillows, dick pills, and reverse mortgages.
If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elon thinks he's laying off.
> In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking.
This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than worsen, this is no longer the case.
Twitter has lost a lot of revenue recently and is now in a lot of debt, although that might not have had time to sink in. In real terms it is in a much worse place, and remember that betting on a success is something you'd do as an investor, not as a rank and file employee.
I'm not sure. I think I'm a good employee, but I like job stability. If I'm working somewhere with layoffs, I'll likely go somewhere without.
There also isn't a measure of employee quality, and layoffs are often not about the individual (e.g. Amazon cutting robotics). I've had no clue how likely I was to be laid off during most points in my career.
I've found the opposite. I almost never see low performing employees fired outside of a mass layoff. In every layoffs I've seen 10x as many people were fired as quit. So you lose a bunch of low performers involuntarily, and a few top performers both voluntarily and involuntarily and that leads to the average quality improve.
My first job out of college had layoffs 8 months in. I was called into a conference room, connected to a speakerphone with the rest of my team, immediately told that we were all laid off and that I had some paperwork to sign in my managers office.
I was given a 33% raise, moved to a salaried position, and got a $5000 cash bonus 10 minutes later.
I had mastered the new system we were implementing and converted and supported the remainder of the customers from the legacy system. Shitty support of the product my team of now one actually drove revenue as the customers would pay higher support prices because of the shitty support I offered.
The service I offered was shitty because my job was to meet minimal contractual obligations for multi-decade contracts to persuade the customers to renegotiate the contracts for a product that hadn't been EOL'd. All the while offering onsite conversion services from the legacy product for most of my day job.
What I'm saying is that company is still alive today, they had a pivot point and literally cut 55% of their employees on a single day.
Google wants to beat facebook still, there is one of a billionity pivot points that twitter could take.
Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I shouldn't. Maybe the effect of all that extra money sloshing around in the system was to incentivize hiring everyone to make sure you didn't accidentally get a false negative, and not all of those hires were good ones.
I think this is not too far wrong. I also think in addition that the really competitive job market of the last 2 years and the hiring they were doing inevitably resulted in some highly paid people who were not good fits or good at their jobs. I think that covid wage hikes and job competitiveness is doing a bit of reckoning now as there is finally enough data to evaluate performance relative to output.
OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff, so maybe it is just that Tech companies are saturated and have too much capital for their creativity.
I'm not doing interviews or surveys, but this also matches with my anecdotal experience with non-"tech" companies, who are all still starving for talent. I guess it makes sense, given that tech companies disproportionately pay in stock and tech companies were disproportionately highly-valued during the 2020-22 Covid weirdness.
>OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff
I wonder if any science fiction writer could have predicted silliness like Snapchat having fewer outages than serious government websites because Snapchat is better at hiring engineers. Tech hiring is so bizarre.
The problem with having a lot of money sloshing around is that you can no longer say “no”. You estimate a new feature will take your team a year. But someone wants it in 6 months. “Why aren’t you hiring to meet the deadline?”
My first job out of Uni was a company that ran financial services, logistic fulfilment, and retail. They had at least a dozen large in-house retail brands, several large warehouses with automation that Amazon was probably behind at the time, and they also ran serviced some of the most well known brands in the country - from their website to customer support to logistics. They had delivery networks, networks of direct shipping, vast call centres with thousands of staff. We integrated with the banking system, with all sorts of external companies.
Our total headcount was under 10 000. Most of that was logistics and call centres (taking retail orders). Our IT dept was literally about 200 ish. All our office staff fit within a few floors of a medium sized building. IT was less than a single floor.
I’d say our software, whilst serving fewer users, was completing tasks orders of magnitude more complex than Twitter. We didn’t have much down time, in fact almost zero beyond physical damage to our networks and the once every few years disaster. Our critical support team was a handful of greybeards who spent much of the time playing solitaire. Code reviews were tight and access to production strictly controlled.
What we did have was a very lean culture, carefully managed over a long time. The company was over 100 years old.
I do not understand how, with far more sophisticated tech, companies like Twitter need so many staff. We could have run Twitter with a thousand folks, including sales. From an engineering standpoint a few hundred would do. And most of us weren’t great engineers. There were a few very experienced freelancers, though. We all worked 0900-1730, but we worked 7 full hours every single day, every minute of those 7 hours. No beanbags. Most of that time was sat quietly coding in a very cleverly designed open plan office (no cubicles) that was like a library.
Looking back, I think the key insight was that we were very cost driven and absolutely kept everything as simple as possible. We had all the usual project overrun problems, usually because things turned out to be more complex than previously imagined. But it baffles me when I hear engineering head count at places like Twitter. If you had given us 1000 engineers, we wouldn’t have known what to do with them.
It is amazing how quickly complexity comes about to serve complexity. Like at $CURRENT_JOB we split everything into a few dozen microservices with their own DBs. Turns out that we need all the data in one place to make some decisions. Shocking I know.
Cue a team of engineers working for a year+ to build a complicated data pipeline to bring all that data back together into a graph DB (not a clue why a graph DB) and build a DSL to write the code to make these decisions.
That's where the engineers go. It's complexity that is introduced setting off a chain reaction of increased complexity to account for the complexity introduced. Then you need complexity to account for that complexity and so on.
Not that it's all self inflicted wounds. Running a site that counts 1/10 of humanity as active users is a hard thing to do
A friend of mine used to work for a private equity firm that had about a hundred C++ and Java developers working on a stock trading and prediction system. He quit and rewrote the essence of that system in F# single-handedly.
If you watch interviews with famous developers like John Carmack, they'll mention that working alone scales to about the equivalent of 5x developers. That is, adding 1-3 extra people might slow you down because of the overheads of communication and coordination. It's only around 5+ in a team that there is a definite advantage.
But what are the chances of putting together a team of 5 rockstar developers that all agree on language, style, and vision? Basically zero. So you have to settle for mediocrity. Popular languages, simple approaches, established design patterns.
If you're an experienced "rockstar" developer coding by yourself and use a fancy language like F#, you can outperform a team of 10+ people. If you're replicating a system you've seen already, 20-50 might not be out of the question, especially if you're smart enough to avoid "tarpits" and instead rely on good quality libraries and CotS components like databases, PaaS, and the like.
It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team.
Organizations tend to bloat.
Random, rapid cuts might not be the fix here, but headcount was too high.
Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up".
Or I tell my boss "code tends to take longer than estimated to deliver".
I wouldn't be given lots of promotions/bonuses with that outlook.
We're talking about a Twitter CEO with $30m/yr+ in total compensation.
There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
We, the people on the outside looking in, have to be missing something.
> There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
> There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
Why not? Why would those board members have ever called him out? The more they're paying him, the more they can pay themselves too.
Yep, my understanding is that cronyism between the executive and the board is a huge issue driving CEO pay. (Boards can also just be out to lunch.)
One way to fix this is to make corporate takeovers easier. This sort of cronyism ends up being a con on the shareholders (and the con is especially easy to pull off if the "shareholders" are passive index funds that don't pay much attention to their holdings).
By removing legal protection on corporate raiding, the board+CEO have to worry about activist investors who ask inconvenient questions like "why are you paying yourself $30M a year instead of giving shareholders a dividend?"
This is a appeal to imperfect authorities that make mistakes all the time. By that logic, Elon was able to convince banks to loan 14B to acquire and gut Twitter. There was no way the staff and shareholders of those organizations, much larger and more proven than Twitter could have been so complacent and ignorant to make such a mistake.
Elon himself is a CEO with a better track record than Parag, why is he not more legitimate if we are pretending leaders can't make mistakes
This is exactly what happens. It's also why CEOs buy other firms even though research shows the majority of takeovers fail. Just pick up any management book in the last 20 years.
There's a clear correlation between org size and CEO pay, even when not in the company's interests. Some CEOs and managers also love the power. Politics is also at work.
> Or I tell my boss "code tends to take longer than estimated to deliver".
I've used variations of that phrase throughout my career. I'm very well paid, partly because I account for this and make certain that other stakeholders are aware.
> We're talking about a Twitter CEO with $30m/yr+ in total compensation.
Jack, who was once the CEO of Twitter, has said that they hired too many people too fast.
Elon, who is "paid" way more than $30 million/year to run several companies, also apparently thinks it was very bloated.
I'm not necessarily saying the billionaires are right and the millions are wrong. I'm just pointing out the clear counterexamples to your appeal to authority. I honestly don't understand why you would be so surprised to find out that millionaires also make plenty of mistakes. The Twitter stock price to a degree is one indicator that the Twitter executives were making mistakes.
Has no company ever been mismanaged? Have they they ever grossly misallocated funds? The answer is of course yes, that happens all the time. Corporate leaderships are not not infallible.
>Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"?
For the same reason that colleges and universities have seen their administrative bloat skyrocket at 10x the rate of student enrollment. Administrative bloat inevitably creeps into all large organizations. Many of the people in the trenches making hiring decisions weren't considering the overall financial performance of Twitter as a company. They were making hiring decisions based on what was happening in their own department, or how that decision would help advance their own agenda, or increase their budget, or increase manpower on a favored project. When you further consider that many at Twitter openly conceded (and in many cases, bragged about) that they viewed their role at Twitter as moral arbiters of society, crucial to policing the discourse of the public, it is not hard to see how enlisting as many true believers as possible to the cause would be seen as desirable, regardless of the larger financial implications.
Indeed. I watched a 30 headcount company that was quite productive getting ballooned up to 170 people after an acquisition. You won’t believe it, but productivity dropped _sharply_. Processes have been introduced that emphasized safety and risk mitigation so heavily that whole product lines are scratched before they can be released. Judging by products and improvements they released within the last year, you could be tempted to think they just stopped working altogether.
And the leads installed by the acquiring company seem to be highly intelligent - the company continues helicoptering in their brightest minds to fix this mess. It just so happens they are forced out every 12-24 months because it’s almost impossible to deliver anything there.
For example, last year they introduced SAFe. That caused an exodus which basically equaled a brain drain (still ongoing). Most people I valued for their straightforwardness and ability to cut through BS there have left or are leaving. The person pushing for SAFe introduction has resigned already.
The founders of that company have all left ofc, and are basically trying to restart the same company (or variations) with their own money. And their products are already picking up speed, becoming threats to the original one.
> These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
Come on, just look at the tech industry. When rates were low and stock prices kept going up, "headcount" was used as an indicator of future growth. Grow headcount, investors are happy. After all, the promise of tech stocks was "growth". Usually you're not looking to cut costs until you think growth is over. Of course, Twitter was a dog and did nothing useful for years, no innovation, no new products, nothing. But tech investors definitely saw rising headcount as a good thing...
> These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
Wealth is not a valid indicator of ability.
I'm not judging the execs and board members individually but rather questioning your assumption. I have read you mention "supposedly", yet it can be read as a rhetorical term.
I've worked in multiple financial services companies where management is incentivised to be as ruthless as possible and they are always overstaffed in areas and understaffed in others. I've been in teams of 10 people that could be staffed by 2.
Hiring often isn't done because of current requirements. Senior execs come and go and with them so do strategic objectives. You accumulate people and they're often not laid off when the thing they work on becomes redundant. Large scale layoffs are awful for morale and usually only come after a 'crisis' occurs.
Now the systems are stable but human workers either be sick, leave, or die eventually.
Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for the human workers unexpectedly not working anymore. Have a multiple workers doing the same things improve stability.
Sure, it's inefficient in terms of money. But alternative is one sick important employee catch a COVID-19 and die lost the knowledge of the system. Documents doesn't solve it because you want the manual operation available right now rather than a few months later when replaced workers learned from the documents.
> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…
>The only reason people work is literally for money…
Well, yes, but also that's super reductive.
The primary reason people work is literally for money. The specific _place_ people work is for things that interest them, engage them, challenge them, provide a path for continued growth in their career, or contribute to a common good they believe in.
And I would argue that it's not necessarily a given that people will be "more engaged and more excited about their work if they were paid more," but instead, may be willing to stick around in an unengaging, unexciting environment _longer_ when they're paid more.
That might be true for you, but way back when, I took a course in programming and thought it was the most fun I'd ever had. Couldn't believe that people wanted to pay me to do it. Spent the next 40 years making a great living. The only jobs I disliked was the ones where there were "glass bowl" bosses and co-workers. Problem solving was always fun. Not saying I didn't enjoy being paid well, but it wasn't the major consideration in whether I wanted to stay in any particular work environment. Here we're talking about the potential for huge schedule and technical pressure and low morale. A whole lot more money wouldn't keep me in those circumstances.
> The only reason people work is literally for money
Theories like Hertzberg exist for a reason and show that money isn't even close to the most important thing for people to work for. If the money is good enough, there are many other factors that are way more important. Not enough money is a reason for people to leave, getting more money is hardly ever a reason to work harder. The possibility of getting more money if they work harder is a good motivator though, once they have it that stops.
This really only works in situations where people are typically underpaid and need to get up to some living standard.
In most software engineering jobs, people have enough money to live an enjoyable life. At that point, having more money is not really such a strong motivator.
I've found that to be absolutely not the case, at least for me. If the job is super disorganized, boring, stresful for stupid reasons or just plain not interesting for me, no amount of money will make me engaged or excited. I've been in such jobs which paid a lot of money (for me) and the result was golden handcuffs - I didn't want to lose the money, so I was basically pretending to be way more engaged than I really was. I suspect majority of FAANG employees are like that - the money's just way too good to not put up a charade and keep milking the cow.
Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post.
Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. As always, the last 2% is a hard problem and business doesn't really want to deal with a such unreliable platform. You wanna onboard big advertisers which potentially spend $100M ARR? Then you need to assign a dedicated account manager to handle all customer escalations. PMs then triage and plan their feature requests and later engineers implement it. Which all adds up.
And they also uses your competitor's product, like Google, FB, TikTok etc etc... Twitter is a severely underdog here, so you need to support at least a minimal, essential subset of features in those products to convince them to spend their money on Twitter. That alone takes hundreds of engineers, data scientists and PM thanks to modern ad serving stacks with massive complexity.
Yeah, it ultimately boils down into a simple fact that it's really hard to take other folk's money. You need to first earn trust from them. They want to see if your product is capable of following a modern standard of digital ad serving for now and foreseeable futures. Twitter has spent lots of time for earning trusts and the original post is one evidence of such efforts. And this usually needs more man power. You might be able to do that in a more efficient manner, but I don't think that's as simple as firing 75% of your entire headcount.
> Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies.
This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outages are short and far between, so they never actually cause real issues. This is obviously costing Meta/Facebook a lot of money, but allows them to be an essential service. So essential in fact, that every major news outlet in my country sends a push message as soon as Whatsapp is down.
If Twitter wants to be a comparably important platform, they need that same stability. And Twitter, for me, is very much the best place to stay up-to-date on any current event (in near real-time). Reddit used to be pretty good with Live, but that's pretty much died (and was mostly a summary of tweets anyway). I really hope Twitter survives Elon, because I don't know of an alternative right now that has the same value in this use case.
I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things.
What Twitter is doing is to scale down first, focus on the product, and once it gains traction, it definitely can scale up again. I don't think it will hurt the product very much.
> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.
I don't think you have a good understanding on how those companies are growing and scaling out. Don't take growth for the granted. "Right product" or "Right technology" won't give you that. It only comes from solving thousands of very specific, never-ending customer problems. If you do B2B, you need to spend most of your time on very specific requests from priority customers. And they are not one, but hundreds of them if you targets $xB business. It's just physically impossible to keep up with a small team even with a very aggressive prioritization.
Still not convinced? Google has a notoriously bad reputation for their customer supports and it's primarily because of their tendency of keeping "inessential headcounts" low as possible. And think about how many cloud customers they lost to AWS and Azure. TK came to GCP and his first work was adding an army of sales and account managers. This almost immediately yielded a rapid acceleration of the platform, although it's too late to catch up.
Scaling back up is really hard though. We had a de facto freeze on hiring (not exactly hiring freeze; more of a headcount cap) just shy of a decade ago to focus on our product. During that time, some of our best recruiters left because they basically had nothing to do anymore.
The freeze worked: we got rid of some products that weren't getting traction and were able to improve the products that did have traction. But the cost of the freeze lingered for at least a year; it reset the hiring pipeline, we couldn't grow fast when we needed to because the limited number of recruiters we had were already overworked, and the limited number of engineers had to balance interviewing needs with their real work. This all happened when my employer was <10% of its current size and pre-IPO, and we didn't even take a headcount reduction.
Twitter is simply at a different scale. 7500 -> 2500 employees is a 66% reduction. Going 2500 -> 7500 is a 200% increase. Recruiting is likely totally gutted, and the current 2500 employees have to support systems previously maintained by a 7500-person company. If they decide they need to grow, it'll have to restart at a snail's pace, and they'll have to make sacrifices on feature development or stability along the way.
Edit: for what it's worth, the fastest way to regrow back 200% is to rehire the people laid off. But, given that I happened to interview earlier today an ex-Twitter candidate who didn't make it through the Elon snap, that route is rapidly closing up.
It is not only hard, it also may or may not work. It is the same process Twitter has already went through years ago. I have simplified the issue and talked only about the product. I don't disagree with you.
> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.
I mean this is just wrong. Companies are always under pressure to cut costs (employees) and it is always talked about when quarterly results are posted. Look at how the market reacted to Facebook's latest results and then again what happened when they laid off thousands of staff.
This is not a trivial task. With such a heavy reduction and how entire teams have been completely decimated, there will be a lot of lost knowledge. I'm sure there will even be cases where the people who stay don't even know what knowledge was lost.
All that puts Twitter in a very risky position, specially in a product of such complexity that does a lot of things in-house. It shouldn't be underestimated.
The headcount at WhatsApp in 2013 was somewhere between 50-100, at which time they were servicing approx 400m MAU, which is more than users than Twitter has been able to boast for most of their existence.
Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had < 100 software engineers. A few short years later and they were re-using rockets, a feat many people had thought unlikely/impossible and requires some hardcore software eng.
Not surprised Elon Musk thinks he can run twitter with a skeleton crew.
1. WhatsApp was more reliable before migration to Meta's infrastructure.
2. WhatsApp didn't have e2ee back then. The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
My point 1 was specifically phrased to clarify how it could have been more reliable before migration to FB because it did not have to deal with the same load back then, you said nothing to show it was not a correlation.
Your point 2 sounds like there were additional factors that could have influence the reliability besides the load (they didn't simply migrate to FB infra but also switched to Signal).
> The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
There's push-notifying millions of devices within seconds after a celeb or a major news source tweets. There's tracking view and engagement stats on that in realtime. There's making sure a tweet is not available to any of those within seconds after it's been deleted or moderator. There're separate back-office apps for moderating that firehose of content. And that's just what I can see from the outside. An e2ee instant messenger with size-limited chat groups doesn't even come close.
Please don't say "just stick a CDN on top of it and you are 99% there", it's embarrassing (and not to twitter). This will maybe get you 80% there if your goal is "a microblogging platform" but not even 20% if your goal is being both the go-to news source and shitpost forum for people worldwide reliably working even in sensitive times and emergencies. Twitter used to be a microblogging platform back when it had much fewer employees and you'd see a fail whale regularly even as it had much fewer active users, in recent yeas it's a completely different beast and saying increased headcount is unrelated is amusing.
WhatsApp has scaled less than 10x since acquisition. They used to handle ~3M open TCP connections per server, and as a result could run their entire operation with under 300 servers.
The push notification argument is also overstated. Sharding and fan-out solves the burstiness. And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter. Besides, these days the push notifications go through Google/Apple servers anyways to reduce the number of open connections needed on the phone side.
Then there are DMs. They are per person so CDNs don't help much (just static assets), but also they shard basically perfectly. So, shard them.
Which in the end leaves the user feeds. Designed correctly, sharding would work extremely well, and what doesn't work could be handled by caching closer to users for those 1k most popular accounts.
Honestly, with the correct architecture, languages and tooling, it could be handled by an experienced 50 person dev team plus another hundred in ops. Obviously Twitter doesn't have the perfect setup, so maybe an order of magnitude more? And if you throw a bunch of subpar engineers and tooling at the problem, nothing can dig you out of inefficiencies at this scale anyways.
And no, I'm not wildly optimistic here. StackOverflow still runs off of 9 on-prem servers [0]. I've seen message queues that can give 200M notifications per second on a single machine (written in C++, for HFT). This stuff is hard yes, but throwing more bodies at it doesn't help past the point your fundamentals are solved.
> WhatsApp has scaled less than 10x since acquisition. They used to handle ~3M open TCP connections per server, and as a result could run their entire operation with under 300 servers.
They switched to a new protocol and grew from 200 million to I guess about a billion users since 2013. If you believe a team of 50 developers could deal with this and not cause extensive downtime and service disruption along the way I pray you never ever manage software engineers.
> Sharding and fan-out solves the burstiness.
Great, at least it's no longer "just add CDN to solve 99%" here;)
> And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter.
Yeah again WhatsApp has many users but as an engineer you just don't ever have to worry about delivering a message instantly to more than 32 people (512 as of this year), and you never have to account for moderating any of that because it's e2ee and there are no adverts next to the messages. It's basically dumb pipes terminated by one native client. Twitter has to maintain a mix of automated and human review of all UGC and is accessible via extensive APIs and search engine indexed web app in addition to native client.
> Then there are DMs
Let's ignore Twitter's DMs, even without them it's far more complex and demanding than an IM app.
> StackOverflow still runs off of 9 on-prem servers [0].
Yeah, and SO maintenance page or read-only mode is up about once a month and lasts dozens of minutes. What are you even talking about now bringing up a niche programmer-oriented help forum for comparison here?
You may be stuck in the times where Twitter was a RoR-based microblogging platform. It's not been that for years.
I do manage software engineers, focusing on HPC (image processing primarily), and one thing I consistently see from people who work with 'classic' web tech is underestimating what modern hardware can do.
This isn't 2005 anymore, we have multiple parallel 40gb LAN, 64 cores per socket and 2MB of L2(!!!) cache per core, and a full terabyte of RAM (!!!) per server. If you program with anything that makes cache-aware data structures and can avoid pointer chasing, your throughput will be astounding and latency will be sub-millisecond. How else do you think WhatsApp managed 300M clients connected per server without having just the in-flight messages overflowing memory, on top of all the TCP connection state?
Things only get slow when scripting languages, serialisation, network calls and neural networks get involved. (AKA "I don't care if you want docker, a function call is 10000x faster than getting a response over gRPC and putting that in the hot loop will increase our hardware requirements by 20x.")
The more distributed your architecture the more network overhead you introduce and the more machines you need. Running the WhatsApp way with less, higher performance servers simply scales better. Just from the hardware improvements since 2013 there was no reason for WhatsApp to change their architecture as they grew.
And if you think rolling out a new protocol while maintaining backwards compatibility is hard and somehow adding more people will help, I have a team of engineers from Accenture to sell you. I did this straight out of university, to thousands of remote devices, over 2G networks, with many of the devices being offline for months in between connections. You just need a solid architecture, competent people and (I can't stress this enough) excellent testing, both automated and manual. And the team that did this was 6 engineers, and this wasn't their only responsibility.
1) Their uptime was great. Regardless of what happened to their uptime after that (when they got to >1B MAU), they were already bigger than Twitter at that point, so...
2) Public broadcasts make a lot of things easier because that means more of your workload is relatively straightforward caching (as evidenced by this blog post).
Have you even thought about moderation and all the other concerns that go with this? How does instantly notifying millions of devices helped with caching, for example?
Yeah, and they were even profitable with ~3k employees. Then the hiring spree started and they went negative. Even if there wasn't Musk they would have to let go at least 30% of the people.
It's largely focussed on the event stream behind the core service and data analytics. There's maybe one entry on the main data store and one on search over the last few years.
Not just that. From my personal experience, twitter had to be one of the slowest websites I use. Even in my 2020 mac, it often shows the memory warning in safari. Things take a while to load. And the UX is terrible with having to constantly click to read child comments, having to click on “show hidden replies” etc. I honestly have no idea how a company with thousands of employees and a billion in loss was able to operate such a terribly performing website.
> What exactly where these thousands of employees actually doing
Maybe trying to increase traffic, attract advertisers, add efficiencies. Not everyone is an SRE. As long as your efforts increase revenue by more than you cost as an employee, you are adding value.
It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did.
All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point.
Maybe it was close (in reference to external factors not implied in your very insightful post!) but its amazing how confident people are with opinions on events they have 0 insight into. Everyone knows how to solve the war in Ukraine or world hunger but how rare is the "consensus" (in terms of up-votes or popularity) right... just something that got me thinking. Thanks again for this article. I always love seeing details of tech ops!
A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.
I wouldn’t be surprised. There were definitely a few bugs. I think ppl unhappy with Elon overplayed things though and now it will look silly.
Our expectations were anchored by the idea that Twitter would cease to exist. So very real problems like 2FA going down look trivial compared to the former.
that's surprising actually. I'd expect my sites to run for about a year if i suddenly died. maybe longer. certs will renew themselves, servers will restart themselves. everything is on autopay.
the thing to kill it would probably be api breaks by 3rd parties.
maybe disk space would kill me eventually too.. i don't have auto resizing for my sql database but its got a long runway yet. or god forbid my site grows on its own and the qps kills it.
I mean, twitter is incomparably more complex than your website (I assume you don’t sit over google or Facebook), plus I’m sure elon made people poke at the system (“we only need 20% of the mucroservices”…)
The QRT system doesn't work (if a post says it has 200 quote tweets, you can always open it and never see any), but then again, it never did seem to work.
It's still entirely believable the site will go down and remain down for a week or two at any time.
It's hard to really say whether that was due to something Elon did or if it was just a normal outage from code deployments, hardware failures, etc. Anything that might seem like an outage will be magnified now because everyone will think it was Elon's doing and it will get tons of clicks. The reality is that these huge sites have a dozen small/medium outages almost every day, but almost no one notices.
I would use it as a case in point. You have one group claiming that Twitter is completely on fire and soon to be closed, and the other saying it's never been better and there was zero fallout from rapidly firing thousands of people including a lot of engineers.
Neither of those extreme viewpoints reflected reality accurately. There were significant problems, but they did not take the entire site down or anything. In fact, for millions who did not have 2FA or use it at that point, they did not see any issue. Whereas people that wanted to find an issue could go looking for the movies. So each side was able to find ways to reinforce their alternate realities.
I think both sides are correct. Anyone with experience at the scale knows that if everyone stops working, the site becomes more reliable. Holiday freezes demonstrate this. But holiday freezes also demonstrate the other thing: if left alone for too long the systems start to rot, from memory leaks and cache ossification and other things that usually aren't noticed during active development.
Personally, I doubt that it will be anything technological that ends Twitter. It will be economic. Their advertising revenue has been decimated and their operating costs have never been higher.
The extent of the 2FA issue I saw - which, by the way, was w.r.t. sms-based 2fa login, not other 2fa methods - was a single screenshot making this claim.
That's what I have been trying to tell people. There are literally two alternate realities and the extreme polarization tries to suck you into either one or the other. Because what people don't realize is that worldviews are tied to group identification.
My theory is that the economic problems have been stressing the systems and that makes the "bones" more apparent. The core concept of this country is checks and balances of two opposing sides. I think it was a great improvement over authoritarianism, but you can see that it is definitely not the ideal final paradigm.
"Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now?
>- Appealing flagged tweets isn't being responded to.
Mine was responded to in a handful of hours. Much faster than I was expecting.
>It's only a matter of time before these get abused with no one to fix them.
The "why hasn’t the site gone down" stuff is the wrong conversation. The engineers built a train engine, and this robber baron held up the train and forced the engineers off. Inertia will keep the train going for a good long while because they did such a good job of building the train.
At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job.
The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass.
Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum.
America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired.
Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.
I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!"
Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment.
Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations.
The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.
It's a form of progressive ideology that is largely based on symbolic actions and relies on mechanisms like shame and shunning, struggle sessions, etc.
So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough.
It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.
He doesn’t understand shit, you don’t fire that many people before even having as crude of an understanding as was seen on that whiteboard. Also, people working on embedded systems for rockets know jack-shit about a system serving petabytes of data.
I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is
"the good times are over." Things start to become brittle and/or not get the attention needed, contractors are brought in because they are cheaper and/or bits get outsourced to the cheapest bidder... the professional care and attention like the author clearly brought just starts to shift over time. Consultants like me are brought in to diagnose what's wrong - the good staff could write our briefs, they know what's going on - and generally we slap a band-aid on the problem because management just wants to squeeze whatever value they can out of the assets rather than actually improve anything.
And that makes him infallible? If anyone else but Musk bought Twitter and started doing what he did they'd be the laughing stock of the financial world. His reputation is the only thing keeping him afloat (in this specific case, he might still do great with SpaceX and Tesla but so what? People like Jobs, Buffet, Gates etc. also made significant blunders at some points in their career)
Any optimization he might do is unlikely to offset the cost of servicing the debt he offloaded to Twitter's balance sheet after the acquisition.
So at 5% interest the yearly cost is 650 million, even at some obscene and unrealistic payroll per employee (including not only developers), like let's say 400k he'd have to fire over 1600 people just to balance this out. There is no way to spin this positively. Also due to no more RSUs cash comp for remaining employee would need to increase.
Add to that the the loss in advertisement revenue do to obvious reasons.
Taking these facts into account could you please explain how is Musk NOT a liability to Twitter? He basically spent 31 billion (plus 13 billion he offload to Twitter) just to buy the trademark and the user base (and some proportion of advertisers...). From a financial standpoint that's an objectively terrible deal. Musk might have other goals and that's great but in no way this is comparable to Tesla or SpaceX.
>But he did change the world with his companies already, didn't he?
History is full of examples of extraordinary successful people who get blinded by their own success and start thinking that they are infallible. This ussually results which in pretty unhinged behavior when they get older. Musk does not seem like the most grounded person ever so he's likely at a much higher risk of this than almost every tech billionaire I can think of.
also: History is full of examples of extraordinary successful people [period].
Maybe somebody with Musk track record can be seen as more credible than some random "qwytw" on the internet? I mean incentives are on his side, aren't they?
For one his famous. It's his toy, so he won't let it disappear into irrelevance without fighting for it. There are not many people currently alive with comparable track record, being recognized world wide, with deep pockets like him.
What if in few years time from now people will be stepping on each other writing posts and books on how they really knew all of that from the beginning, that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
To be honest, I don't even think anything extraordinary needs to happen for all of it to work out just fine.
For all people saying technically it's not just rendering short messages - well it is just that at the end.
Look at stackoverflow - most people don't realize it's just single monolith application running on 6 servers - all of it.
There are still thousands of people employed there, which feels like too many if anything.
Did I misrepresent any facts or figures? Did I lie? What does credibility have anything to do with anything otherwise? It's an objectively bad deal from a financial perspective, if you have any arguments against that you're free to share them.
Twitter might do "fine" as a platform. Sure, why not... Does not change the fact that he grossly overpaid and that the 13 billion in additional debt will be financially crippling for the company. He might sell some more Tesla stock to offset that. I mean it's his money, he can do whatever he wants (unless the SEC stops being a complete joke).
> that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
Right about what? Even Zuckerberg seems to have a clearer vision about the Metaverse than Musk seems to have about the future of Twitter. I might be wrong. Care to enlighten me instead of talking about some random tangentially related things?
> There are still thousands of people employed there, which feels like too many if anything.
Sure, can't really disagree. However is it better for a business to have an additional 2000 employees or so who are possibly superfluous or to fire all of them and spend ~600 million or thereabouts servicing a debt incurred for no reason? It's a pretty straightforward you chose to ignore for some reason.
To play devil's advocate for a bit, are you certain he's solely responsible for the successes of SpaceX/Tesla, or that those companies succeeded in spite of his leadership?
In case it isn't clear, I agree with the parent comment. In the same tone as the comment above the parent, you could also write "Thing is - Elon Musk knows a thing or two about sexual harassment lawsuits. It's inevitable that he will have another at Twitter."
Stop idolizing billionaires.
>Musk has successfully run companies in the past and is running two extremely successful companies right now.
Is he actually running those companies, or is he posting selfies of whiteboards at 1:30am showing architecture diagrams of Twitter?
In these markets, after he was nearly forced by courts into completing a purchase deal for Twitter after making an overpriced offer, I'm genuinely curious to see how he pays the ~$1b in interest per year on the debt to purchase Twitter.
No, people are idolizing Musk because he is a billionaire.
People who obviously know absolutely fucking nothing about coding or manufacturing or running businesses or software engineering or cars or space or rockets or emerald mines are continuously defending Musk, and their only justification is that "he's rich, so he's obviously smarter than you!"
> No, people are idolizing Musk because he is a billionaire.
No they are not. They are idolizing him specifically because of the success of Tesla, SpaceX, and PayPal.
> "he's rich, so he's obviously smarter than you!"
Nobody is making that argument. Its usually, “look at the success of the companies he leads”.
You can make an argument about whether or not he should get the credit he does for the success, but nobody is idolizing the wealth.
None of these people are going to defend the great insight of the winner of the last powerball because there is no repeated pattern of long term success.
The US has been formed religiously by various congregations that descend from Calvinism, which teaches that hard work earns you the grace of God. In modern times, this has morphed into Prosperity Gospel which teaches that if you are loved by God, you will become materially successful, as well as the flipside; if you are materially successful, you must therefore be loved by God.
The billionaire worship is simply the secular version of this faith. X made billions of dollars by owning a business that does Y, therefore X must be an expert on Y, and businesses in general, and is a genius, and is a morally superior person. Otherwise X wouldn't be a billionaire, because - horror of all horrors - the world is just, isn't it? It couldn't be the case that X just got lucky, can it? Surely, X has earned their billions, right?
You’re still fixating on the wealth part, which people on this site (at least the arguments I see upvoted) do not.
People are “worshipping” Musk because of his repeated success in making successful large innovative businesses. The wealth he has is a product of that, but the deference is to the success of the companies.
Funny enough it would be hard to find anyone with more CEO experience than Musk. This doesn't mean he does all the management, often the work of these guys is to find the right people to do the management for them.
SpaceX is subsidised (and so is Tesla) and automotive industry was already in the process of electrifying. Tesla just put pressure to speed it up. Haven't checked but it would not surprises me if Formula E predated Tesla.
Winning government contracts over other competitors is not the same thing as “subsidized”. Otherwise Hilton is subsidized because government employees stay there sometimes.
I get the impression that some government contracts are used as stealth subsidies. And SpaceX, Tesla, and Starlink have benefited from plenty of actual subsidies, too.
To be clear, I'm not implying that's a bad thing. These companies seem like an example of subsidies working as intended, i.e. driving innovation.
Can you give specific examples? Diluting the meaning down to “the government is also one of this company’s customers” just makes it meaningless. That will cover nearly every large company.
In January 2020, Toyota passed the milestone of more than 15 million self-charging hybrid vehicles
If every vehicle sold by Tesla since its inception up to the most recent Tesla sales figures released for Q1 of 2022 are accounted for, there are over 2,645,000 Teslas on the road worldwide
Sure, cheap hybrids. Nobody is buying them because it's their dream car though, it is just affordable and cheap to run. Tesla makes cars that are desirable.
He also doesn't make decisions like this from monologues in his head - he surrounds himself with people having deep expertise and makes bold decisions based on feedback; if something doesn't make sense, they'll stop doing it the next morning. In "normal" corp not only it'll be there 3 years from now, it'll also grow fat around it and spawn other idiotic initiatives with little to no (negative?) value.
Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies?
Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.
Another company he runs, Tesla, is famous for being one of the few car companies that makes software for their cars that people actually want to use.
If a citation is needed, it is a citation to explain how anyone could possibly believe with that track record that Elon Musk doesn't know how software is built.
Knowing how to run a software company and knowing how to build software are two things. The first thing usually implies hiring the right people for the second thing. I’m sure Musk knows at least a bit of that, but everything we see with Twitter is that he’s axing those people at a rate that you can’t keep up hiring at. So there is some obvious disconnect here and I don’t think “Ah, he knows what he is doing “ is a sufficient explanation.
The Zip2 application ran on a single PC. Technology has changed a lot from the early Java days(I think what Zip2 was written in). Perhaps this helps explain some of his public mistakes in understanding the codebase of Twitter.
Look, I actually do agree that Gwynne Shotwell has been a key component in making SpaceX successful / curbing Musk's worst impulses / etc.
But don't you think it's strange that (assuming so much of SpaceX's success is owed to Shotwell) nothing like what SpaceX has achieved was achieved at her previous gigs? But then she goes to work for Elon Musk and suddenly...
Regardless, the point I'm making is not "Elon Musk wrote all the software at his various companies himself" but rather "Elon Musk founding / running / exiting a variety of companies in very diverse fields but all of which are good at software is a very strong indicator that he understands software and how it is built".
Meanwhile, the "he doesn't understand software" corpus of evidence basically amounts to "I think he's doing a bad job of running Twitter in the handful of weeks he has been there. I think this even though I have very limited insight into what is actually happening at Twitter, besides what is being fed to the media by disgruntled employees who probably hated Elon Musk before the acquisition and their subsequent firing".
> SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.
Not to be dismissive of landing an orbital rocket, but that's mostly a feat of aerospace engineering and control theory. On the software engineering side of things for SpaceX I think their work around CFD would be the more impressive feat from a computer science perspective.
Elon just made sweeping changes that cratered their ad revenue and took an axe to engineering teams all over the company. He did this in the lead up to the holiday season which has been the most profitable part of the year with advertisers pouring a significant portion of their yearly ad budgets into ad buys. Even if he was 100% correct that all of this was bloat, it was a monumentally stupid idea to make these changes now of all times. I would have expected him to at least have enough of an understanding of software engineering to know that deprecating systems and cutting out bloat needs to happen carefully with the talent onhand to be able to back out changes and manage the transition seamlessly. Fundamentally the risks involved are more financially impactful than the price of keeping headcount around for another year to do it safely.
Twitter hasn't even had a major incident yet and just the revenue loss from advertisers pausing ad campaigns probably already offsets all the savings they were hoping for in 4 - 6 months once the layoffs finally start bringing down staff costs.
Look at his claim that Twitter is slow in India because of poorly batched backend calls [1].
He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random.
Stop repeating this, because Elon had the gist of it correct, if not the specifics.
a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time.
b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
c) Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
Elon accidentally conflated the network path of (b) with the performance issues caused by (c).
They're both bad! They both need fixing! The fact that Musk mixed up doesn't matter.
> Much of that is because of hundreds of client-server JavaScript requests on first load.
Isn't this not true though? Based on the responses to his Tweet only one GraphQL request is made which batches the whole thing.
> Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
But what does this have to do with India? If the slowness is due to calls between microservices hosted on (from my understanding) their own hardware, why would it matter if the initial GraphQL request comes in from India, since all the rest of the backend requests would be the same?
> his Tweet only one GraphQL request is made which batches the whole thing.
Literally bang in the middle of an F12 network trace showing dozens of requests, mostly JavaScript. Not JSON RPC, per se, but still an overhead taking up seconds, especially on low-end Android devices.
> But what does this have to do with India?
Everything. Nitter uses 40KB of code & data to display the same content that takes Twitter megabytes.
People in subsequent threads have pointed to the gigabytes of "data" usage by the Twitter Android app, which is just absurd for an app that displays primarily text.
It's a bloated pig of an app that performs poorly on 5G networks in the United States, and hence terribly on slow and overloaded networks separated from Twitter's data centres by an ocean.
"It works fine for me" says the engineer connected to the WiFi in Twitter HQ.
> Literally bang in the middle of an F12 network trace showing dozens of requests, mostly JavaScript. Not JSON RPC, per se, but still an overhead taking up seconds, especially on low-end Android devices.
I'm pretty sure that's my point? How are you equating "dozens" to "1200" (or even "hundreds" in your last comment) as though Elon is in the right ballpark?
As for the rest, I don't know what your point is, where did Elon say anything in that thread about how much data they were sending? He's quite literally responding to someone saying it was the amount of JavaScript being downloaded and telling them they're wrong, it's the "1200 RPC calls". I'm pretty sure the point he was attempting to make is that it's slow due to thousands of serial "RPC" calls made between the app and Twitter (which will have a higher RRT to India) but as you've demonstrated that's not what it's doing.
I'm not saying Twitter is well designed, I'm just not pretending like Elon somehow had the right idea when you've had to stretch it this far. If he had a proper technical understanding then we wouldn't have to have such a discussion of what he "actually meant".
That's correct, I am working in mobile world since 2005 and the latency of mobile networks is the most bad thing when you build mobile apps. The more requests you do the more latency you introduce to the user, especially if it's fetching data for UI, even if you do it asynchronously you have to wait for the data to update the UI.
> Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
This is not true at all. I have a look and it's not that there are many REST requests it's that the requests send back a load of data. For Example the timeline sends back a 1MB JSON response. It's heavily gzipped so comes in at 100KB over the network but it takes quite a while for the whole request.
In places like india the internet is slow and many people are using VPNs. I emulated this and I can see that over VPN on a slow internet connection it's like 10 seconds to load the home page.
He was right on the details of the two different issues, because those were explained to him.
But he conflated two things that he didn't really understand and reached a completely wrong conclusion. He's shared that conclusion with confidence and never admitted he was wrong. That's a fatal flaw for someone who holds a Chief Engineer title at a rocket company.
He's a CEO who is has been incredibly successful at selling a vision and getting people to do what he wants. He is not an engineer. He should stop role playing as one.
Yeah he is so smart, he locked himself out in firing spree during some manic phase and then had to beg people he just fired to let him in. If anything, this twitter saga showed serious cracks (that were there before if you looked closely) in his persona. Vengeful, petty childish behavior was there before (ie Musk accusing main Thai cave rescuer for being a pedophile, because he refused his submarine concept? Plenty of similar stories)
Its a matter of approach - if you focus on all his success, you can continue worshiping him as some capitalist messiah. If you focus on bad stuff, he is a fucked up mess to be polite, a bad parent and husband, an even worse human being in general, utterly horrible boss, person with deep mental issues that aren't clearly tackled in appropriate ways.
The truth, somewhere in the middle of those 2 extremes, still ain't that nice IMHO. He burned basically all good PR capital be built over the years, basically nobody sees him now as good person, just another ultra rich a-hole like Bezos that needs to be kept in check. As he will find in following years, people's opinions trickle down to cash flow. If I would be buying an electric car right now (which doesn't make much sense for another decade to me), Tesla would be off the table for example.
I don't know man. His last pure software company was PayPal and they fired his ass from the CEO role and put in a competent CEO (Peter Theil) to right the ship and sell it off before the 01 crash.
Tesla and SpaceX have been wildly successful and he will forever recieve some well deserved credit for their successes but at the same time I think its pretty bad that Tesla (and SpaceX) have had multiple near death experiences and some them could have been avoided if it weren't for incompetence.
I followed the Model 3 production religiously and the amount of dumb mistakes they made even after other industry experts were yelling at them about imminent iceberg collision is just unbelievable. In fact the Model 3 production, with its horribly designed body that he wasn't keeping on top of and Musk's stupid attempts to automate all of assembly should be entirely attributed to him. It was so bad it was worse than even the shortseller's worst predictions.
I guess you can give him a pass for near bankruptcy #1(Roadster roll out) and #4(COVID lockdowns). Not sure about #2(Model S early sales woes and potential acquisition by Google)
All I am saying is that he fucks up A LOT and has thus far managed to stave off the consequences of his decisions but many occurrences have been down to luck and luck is something you should never bet on.
Yeah, but that's not what's happening here, is it? Musk is pretty obviously looking for a corporate culture shift, not turning the company into a cash cow - and for this to work, entropy needs only to work slower than half a year. Which this article argues pretty convincingly is going to happen.
The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wholly unqualified interviewers, but they now also have to handle nightmare features that were built by people completely disconnected from the other side.
I'm not a friend of Elon's, but outside of the flashiness of the whole thing, I don't think his firing spree was wholly unwarranted.
The other day I saw a video of a bunch of people at twitter leaving that have been there for a decade or so. I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
All companies I’ve worked at, startup to FAANG, have tried their best to reduce attrition and retain employees by providing growth opportunities.
Changing jobs is a skill/knowledge net-negative for the employer, and can be negative (outside of salary) for the employee as they have to relearn everything (relationships, tech debt, processes) etc.
Except in these firing sprees it's not competent employees that remain - those people are usually the first to jump ship because they have options and see where it's headed. It's not like they get raises for staying to get a part of the savings from the cuts - usually they get pay freezes.
Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product".
I'm not that optimistic about Twitter long term - never was TBH but this Musk thing is turning to a shitshow and also exposes what it's like working for him in his other companies.
You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue. Data scientists that write a blog post about the most used hashtag or emoticon. Or PMs that we don't need...
I have not been impressed with the average tech person's ability to understand parts of a business outside their immediate responsibilities. There were people on here seriously questioning the need for a general counsel for a company that's constantly under legal assault by every country on the planet. Never mind the long recurring "why does an ad-dependent company need people to manage relationships with the big spenders who expect that?" discussions.
By being a free speech platform without the censoring role of an editor, Twitter does not need to care if Qatar is mad about what is written there. If Qatar bans twitter, they ban twitter.
What is a "legal assault" and why does Lichtenstein have an ongoing "legal assault" against twitter?
The previous point is very reasonable, and I think your response is overly flippant and confrontational.
I've had coworkers hounded by managers from unrelated teams because my coworker wasn't working fast enough on a feature that was important to their team. They made assumptions about how many different tasks my coworker might have had assigned to them.
They were juggling like 3 or 4 fairly important features/fixes for various parts of the organization at the time because our team was (in my opinion) understaffed.
But of course, each of the relevant managers can only see "didn't satisfactorily complete my fix in the expected timeframe", and then perf review comes back and collates that info to "doesn't complete tasks on schedule" without taking a step back and recognizing that this is a failure mode of planning where one engineer was responsible for way too much stuff with no reasonable ability to push back.
At the same time, of course, engineers were getting promotions for doing exactly one thing well, which is as it should be.
There are lots of lessons in here, one of which is very likely "push back on work that causes you to overburden yourself", but I'd argue another important lesson is that you just don't see what others are working on. I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
But while we're throwing ad hominems for the fun of it; I'd rather have a bunch of "bloaters" on my team than the one person who is convinced that they are one of the golden few producing value, and everyone else is a lazy freeloader. I've worked with that type of person before and it's awful. It is rarely the case that they are _actually_ producing an 2-3x the value of other engineers, it's much more likely that they're just reducing morale and causing internal conflict while building some small piece of the pie and assuming that's the majority of the work.
Like someone who builds a button that says "delete my account" and says "I built the delete account feature" while not recognizing that there are a ton of people on the database teams that made the feature possible without performance hiccups or leaving dangling foreign keys.
You are bringing up a completely different problem that's a whole another case than bloat: bad managers that don't realise that some workers are juggling many problems, not just their own. That is very different to people doing a lot of work in secret in an unsecretive organisation, like twitter.
> I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
This is not the case at all in many work situations, a prime example being when you work on a team and all code and work produced is shared. None works on other teams. The code reviewer and hopefully project lead will know, or have a very good hunch, what everyone is producing.
I would argue that bad middle management is another bloat problem, so in essence; you just made another point for how bloat is very bad for an organisation. I wonder why you are so interested in defending the all too common effect of corporate bloat in an economy where its calculated that a large percentage of jobs are useless and there is a trend to collect multiple useless jobs.
>You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue.
Absolutely. I also realize that when a company reaches the firing squad stage it's not a good idea to stick around - better polish up the CV and move to greener pastures. I have no insider information - they could be offering generous (and credible) retention bonuses for critical staff, but I've seen this happen twice before indirectly (I was gone before it started happening) and the people that stayed didn't have a good time, were lied to and left in the end anyway in a worse situation (or waited trough to bankruptcy and had to sue for outstanding wages in first place I worked in)
Depends on if you want innovation or stability. For innovation, people building fiefdoms over decades of political maneuvering is terribly destructive to change.
Very few industries require constant learning for the business to compete so a highly tenured employee likely hasn’t learned anything new beyond minute process changes for 10+ years. Once people are ossified into a role like that, they will do anything and everything to shut down anything that has a whiff of threatening their current role.
That is actually not my experience. I worked at a scientific institute on space cameras, and since projects easily last 10 years from design to launch, we have a lot of older people.
Some were resisting change. Some were embracing it. All in all, I had a great time and gained a lot of respect. I love working in teams with all ages.
Your sentence "once people are ossified..." feels like a generalization to me.
Yes and no. I think we'd all agree that large tech companies have tons of really obvious staffing inefficiencies. There are many teams working on what are essentially vanity projects and many other teams have more staffing than they realistically need.
On the other hand the slash and burn Elon approach seems objectively terrible. Indiscriminately firing most of the company kills morale and is likely to send the company into a hiring death spiral where your good employees leave and you can't attract good talent. This won't automatically kill the product or the company but it's not going to lend itself to big positive successes in the future.
> I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can speed up your work: whom to ask on the "kurzer Dienstweg" aka short-circuitting bureaucracy when needed, personal relationships with people in other departments on whose knowledge you rely (it's one thing if you get a random email asking for some shit from someone you don't know, but I'll always find some minutes to help out someone who has helped me out in the past), all the domain-specific knowledge about the precise needs and desires of your customers...
Attrition is bad for a company as a whole, the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact), and so there is no KPI for leadership other than attrition rate itself.
The only problem is that over the last years, employers' mindset has shifted from regular wage raises to paying the bare minimum which makes changing jobs every few years a virtual requirement for employees to get raises, and so we are already seeing the first glimpses of US employment culture and its issues cropping up.
> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact)
This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in companies like that.
This was basically Google’s entire pitch when they were an early public company. Happy employees == great things. I remember when 60 minutes or dateline did some special about google before 2010 and people were floored by how good the employees got it. However, hiring standards relaxed and now google is slowly rotting from the inside to realize it’s the new IBM.
Long tenure alone is absolutely not an advantage. It’s very easy to have a bunch of dead weight that just looks like they know what’s going on.
So back to your point about experience mattering. That should show up in customer satisfaction, project turnaround speed, success rates, etc. Otherwise it’s pretty meaningless. And guess who already measures those KPIs…
I think this is down to the time horizons each kind of company optimizes for. Mittelstands are usually owned by families and they basically try to build a business that will feed the parents as well as kids when they come of age. Publicly listed companies usually optimize for this quarter. So even if you are looking at the same KPIs, you could get different outcomes just due to the utility functions being different.
> This is false. People invest on many year horizons all of the time when they believe in the company.
The perfect example of why that is not the case at least in Silicon Valley is Google and their messengers and social networks... just how many of them exist(ed)? Four? Five? Each of them was someone's promotion project, left to languish, suffer and eventually die afterwards. Honestly it's a wonder Gmail and Maps are still around, their UI hasn't been updated much in ages despite obvious potential for improvements... I guess the only reason why they are alive is that they generate absurd amounts of data.
> This is false. People invest on many year horizons all of the time when they believe in the company.
Granted, but they usually have the option to sell at many points along the way.
Think about an alternative financial instrument. What if you could invest $X in a public company (and get a higher rate of return because of the higher risk) but you lose the ability to sell until five years have passed?
This is one kind of option that might be worth trying. I want to see more mechanisms promoting long-term investment in our public markets. (Bonds are not identical to what I described BTW.)
The lack of these kinds of options (sure, bonds exist but I've not noticed them used with any notable significance in public companies) is why people talk about the short term quarterly focus of Wall Street.
I don’t understand what the point is. You have the option already to hold on for 5 years+. Many people do this (especially employees with vesting schedules) and they are rewarded for doing it.
> is why people talk about the short term quarterly focus of Wall Street.
This is a false meme though. Tesla was worth more than multiple profitable long standing car manufacturers combined before it ever turned a profit. Investors fixate on short term performance when the company has no long term vision and deriving cash flows is much easier (e.g. a container ship holding company).
> You have the option already to hold on for 5 years+.
The difference is having the option. :) Behavior tends to be different if you don't have the option to sell during a window of time.
In this hypothetical case it might serve as a built-in grace period. Companies that otherwise might have failed if measured and punished quarter by quarter might have time to build a product over five year time horizon.
> Long tenure alone is absolutely not an advantage.
It's also not necessarily a disadvantage. The real argument is that people management is important. Everyone from the new person who just joined to the decade long senior engineer need to be held to high standards. Which brings me to my next point.
The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything. I think people routinely underestimate how important good leaders are. "There are no bad teams, only bad leaders" comes to mind. So far Elon has shown himself to be terrible in this regard. With SpaceX and Tesla he communicated a vision, with Twitter he's communicated very little and just instituted chaos.
> The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything.
The problem was even worse: there was no vision at all from no one, not leadership, not the investors, not the users, what Twitter should be, other than "it is a way for instant communication with feeds". Everything else was completely lacking: what features do people want, what moderation policy should be applied, how does Twitter plan to make money.
The only ones that had at least some sort of vision where activists - the left wing, the advertisers and large parts of the users didn't want Nazis any more, and the right wing wanted "free speech" aka allowing Nazis.
> The reality is most huge companies are majority bloat.
This is true, and in my opinion, true for a reason. And that reason is not "most huge companies are dumb", as opposed to what Musk's cult seem to believe. The reality is, measuring what exactly is "bloat" and precisely cutting that bloat is extremely difficult and firing more than half of your workforce is probably like using a warhammer to do brain surgery.
Well, your alias actually refers to that process. Large organizations gain entropy over time. And entropy is precisly the inability to describe some structure at a micro/nano-level.
But while fighting the entropy can be hard, not doing so is almost certain to be lethal.
If an organism is infected by gangrene or cancer (the more extreme forms of entropy for a body), it may be more realistic to cut away whole body parts than to treat it in place, even if there is risk of sudden death.
It seems to em that this is what Musk is trying. Either Twitter will be gone within a year, or they may very well be sustainably profitable within 2-3 years.
Software engineers at Twitter got used to working on an money losing company for 10 years and being told they’re great at their job. Then they were fired en masse because someone’s called them out on it.
If your company is losing money all this time you are likely to be fired eventually in the real world. Job security in sw world had become so high that no one seemed to expect it. Everyone assumed “sure we’re losing money and the company has no direction” but all is fine.
They all stayed there in their tables working for 10 years in a rudderless company as if it was a government job.
This is factually untrue though. Twitter was making money in 2018 and 2019 (to the tune of ~1.2B/year in net profit out of ~3B revenue, which is a fairly high profit margin) they lost quite a bit of money in 2020 and less in the years thereafter. However, even in the years where they had negative net income EBITDA remained positive suggesting the losses were upfront investments that would be expected to be amortized over the coming years.
The only reason Twitter is in deep financial shit right now is because Elon acquired it in a leveraged buyout and the cost of servicing the debt is estimated around 1B/year.
Yeah, most of the complaints of Twitter was that it should be doing more in the space it commanded, not that it was losing tons of money every year. Though, 1-1.5B in yearly interest payments is going to make that tough going forward.
>The reality is most huge companies are majority bloat.
Something I learned early in my career is that some companies consider this a feature not a bug. As in, they are hoarding talent so that they don't go to work for their competitor.
Twitter didn't start yesterday. There was a time when Mesos was all the hotness and k8s was this new thing that looked promising but wasn't nearly production ready.
Apple is also a big Mesos user, but also moving to k8s.
They were one of the first users of Mesos, but they didn't create it.
> Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica.
> The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers.
The company was moving off it. I wish I could find a twitter eng blog about it. Interesting there's a bunch of not directly from twitter sources about the decision though.
The Mesos detail struck me as a risk for the reason you stated.
Also, the article seems to suggest Twitter only has two datacenters. That seems surprising for the global reach of the company. Perhaps there are other smaller datacenters that are not prepared to handle the entirety of the site’s traffic.
My current thinking is there’s time to figure out how to operate the current system before it runs into issues that would render it degraded for a prolonged period of time. I noticed TLS certs have already rotated for instance. That was my best guess for simple thing that could fail if managed poorly.
Why do people assume that things would break so easily? I'm pretty sure Twitter (ex)team did tons of scalability testing with loads that are 10x or even 100x. Plus random 100x to 1000x spikes to account for breaking news. And all of these are done automatically, constantly, and consistently over the years.
I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.
It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.
I would say fart noises are the comedy equivalent of fart noises, but in any case, I sure do love people getting upset by the comedy equivalent of fart noises. Speaking of which, I would actually pay Musk $8 if it meant he would post a video of himself doing the old underarm fart noise thing and I could see all the fantastic "well I never" reactions from everybody who got upset by it.
>It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into.
What's wrong with the new owner of Twitter using Twitter in exactly the same way that a lot of other Twitter users use it for, memes and jokes?
I find that rather refreshing, actually. It's a variant of eating one's own dogfood.
It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.
He can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.
Is it strange and inappropriate if most people think it is a funny joke and laugh it off? Who goes out of their way to look at Elon posts just to be offended?
His process tree has only produced private and reusable rockets, space internet, electric vehicles, and hundreds of billions in value. But he makes crude jokes like 90% of the men on the whole planet. Clearly there's something deeply wrong.
So if the people HE hired did those things, doesn't that mean HE put the team together, HE found the talent, HE set the vision, goals, and tasks, so HE led them to accomplish these things, along with funding them? It also means HE didn't design the SDS you claim to be defective, almost like such software is difficult to develop.
It is funny. I think it's hilarious and the fact he shared it in public makes it even more so. It's one of his most liked tweets, so a lot of other people also thought it was funny. Comedy is subjective, but if a lot of people think something is funny then it is objectively funny. And it is perfectly appropriate for public figures to reveal something of themselves and have some fun in public. Puritans will always be offended by this kind of sex joke, or by the gays being allowed to marry or whatever, but we laugh at them too, they make it even funnier.
In my opinion what would be inappropriate for a billionaire CEO, and reveal deep flaws in decision making process, and the decision making process of any government official or governing body who subsequently partners with that CEO, is to fraternize with the likes of Epstein after he was convicted, e.g., https://www.nytimes.com/2019/10/12/business/jeffrey-epstein-.... But I guess morality is also subjective so people might disagree with me there too.
Lots of people are puritans, I was talking about the puritans making themselves terribly offended by that meme. If Musk is one too it doesn't mean they aren't, it's not based on some relative scale here. But I am using the lowercase puritan which probably doesn't fit him.
I didn't know that he lead an extremist cult though, not that I have ever cared to find out much about his life or beliefs. It wouldn't surprise me, I think a lot of the ruling class and ultra rich are terrible people who we should not be living under. That said I have noticed the claims of Musk's terribleness and extremism may have a tiny correlation with his unwillingness to toe the Democrat party line. Now quite possibly that's just a coincidence and there's absolutely no causation there, nevertheless if I were ever in a situation where that information would be important to a decision, it would behoove me to take it on its merits rather than on faith.
There's a pattern of diverging expectations here, one is the non-technical/naïve one,
- Twitter is going to go down tomorrow and it's all over. RIP.
The second is,
- Twitter is going to experience a failure cascade over time.
The third is,
- It's all going to be fine.
I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps?
I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash.
From what I can tell, in the past week or so,
- Twitter's copyright system failed
- Two Factor Authentication broke down (it seems to be back up?)
- (anecdata) Tweets have been loading sporadically for me and other people, sometimes we try to open a tweet and it says that it doesn't exist. Happens more frequently with new/recent tweets.
- (unconfirmed) Twitter's managed account backend is behaving "strangely." For e.g., "One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated." from https://www.teamblind.com/post/i-told-my-team-to-pause-our-750kmonth-twitter-ads-budget-last-week-4dnbo1Ft ———— Friends have told me other similar stories
Are these failures symptomatic of a larger problem, or are they well-isolated parts misbehaving? Can Twitter even experience a failure cascade like Arecibo? Can that be paused/stopped?
I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.
Also anecdote, but twitter has historically been pretty unreliable for me.
Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.
Definitely this. I regularly see people complaining about ancient bugs as if they were new after the acquisition. Nope, you just are engaged enough to notice them now.
After the failwhale days, Twitter has been quite stable for me with the only exception being the live-refresh features on tabs of the website that have been open for days (which I don't think many websites would handle well).
There has been a serious degradation in the quality since the acquisition:
- Sporadically loading tweets - I could go on some tweets and refresh the page multiple times, with tweets fading in and out of existence showing "This tweet is unavailable"
- Tweets that quote tweets of accounts you have blocked behave weird in multiple ways. Sometimes it's just showing a "This tweet is unavailable" instead of "This tweet is from an account you have blocked", and a few interacting with them crashed my timeline on mobile, having to restart the app
- On a few occurences, every third tweet on the timeline was an ad
- 1-2 notifications from crypto spam bots reaching me every day. The same thing previously was filtered out quite reliably (I assume), since then it happened ~once ever 6 months
--------
And those are only the things I've seen personally. Yeah, they are no deal-breakers, and mostly sporadic failures, but it very much feels like a service that is degrading by the day.
Perhaps the failure mode is like the Tacoma Narrows Bridge, with Elon as the wind, the undulations his whipsaw changes of mind, and the employees the hapless cars being flung into the river.
1,419 comments
[ 1.7 ms ] story [ 199 ms ] threadmy condolences
https://www.nytimes.com/2022/11/18/technology/elon-musk-twit...
What did you make of Mudge's report regarding resiliency of data-centers?
> Insufficient data center redundancy, 59 without a plan to cold-boot or recover from even minor overlapping data center failure, raising the risk of a brief outage to that of a catastrophic and existential risk for Twitter's survival.
- https://techpolicy.press/wp-content/uploads/2022/08/whistleb...
To be this young and carefree
These kinds of things have formal courtship type periods and legal filings that take months, and then they happen, and then executives plan on rolling out layoffs and cost cutting measures over years, not days or weeks, like the "plan" seems to be here. This really is madness, and I can't blame people for thinking the site might just wobble and collapse at some point, though as the author goes into in very nice detail, we do try and build resilient systems. I'm certainly curious how it all plays out, but you won't catch me making any guesses.
That's certainly possible and I wouldn't know what that would look like, and yep, in theory there are lots of financial rules and regulations regarding public companies that Musk made a bit of a hash of.
I can't agree that Twitter's journalist demographic was mostly responsible for this being big news, though I'm sure that plays a part. Musk brings the circus with him wherever he goes, but this was nuts even for him. And any $44 billion tech buyout would be news without those factors.
I think you hit on the the main point: the reason this is constantly in the mainstream media is because the mainstream media so heavily relied on/still relies on twitter[1].
If this were any other large company imploding in a Musk acquisition we'd see fewer stories about it because it affects journalists less.
--------
[1] It's going to be especially funny if journalists complain that "#LearnToCode" type of tweeters aren't getting banned, and get told in response to go off and create their own twitter.
No matter where in the political spectrum one may lie, it's always satisfying to see a group being fed their own lines back.
Although I haven't worked at Twitter, if it is anything like the places I have worked at, the small issues get ignored because nobody makes their career on things that don't show up on investor / shareholder / upper management radars. If it can get papered over so it ends up being someone else's problem in the future, what actually gets worked on is flashier things like redesigning the timeline AGAIN so users spend more time on the website or (in the case that caused me to work for small startups again) spend three weeks deciding on a super trivial design decision with 12 stake holders.
This seems to be more true of Mastodon than Twitter.
I can't imagine any self hosted Mastodon instance staying up longer than twitter.
Second, Mastodon runs on open protocols. That has good and bad points - for example, it won’t grow as quickly as a project with huge corporate backing - but it does mean that there’s a more direct link between the community and its longevity. Twitter isn’t just flailing because Musk is doing management by bong rip but also because he’s desperately trying to get out of a financial hole. Open source projects have different kinds of financial challenges but they’re never on the hook trying to fill a hole measured in billions of dollars, either. Given the number of communities older than Twitter I’d say it’s far from proven that Twitter will outlast anything.
- ghost towns with little to no one there.
- didn't open registrations.
- completely dead with ssl invalid certificates with expired domains.
So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above.
There is no monetary incentive to keep a mastodon instance running and we both know that begging for donations doesn't scale.
I don't think "begging for donations" needs to scale, if instances get too large to keep running then smaller instances should (and do) split off, they can still talk to each other after all.
There's an actively maintained public instance list here: https://joinmastodon.org/servers
I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.)
Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" and change. For the preservation of their own self-interests, of course; much like the company that makes physical objects too robust and runs out of customers, planned obsolescence and the desire to change things and justify it so they can be paid to do something are still there.
It's possible to make things which last. Unfortunately, much of the time, other economic considerations preclude that.
The fact that most software doesn't continue to function even on the same platform, and on the same hardware, is a massive indictment of the software industry's standard practices.
An application running on a single platform, self-contained and with some basic failovers such as redundancy (2+ machines running the same application), etc. should have ridiculous uptimes.
A distributed and complex system with interdependent components, under variable load, with different capacities for subsystems running across some thousands of machines will, inevitably, encounter some unforeseen state which degrades the system as a whole. It can be a small component that cascades a failure mode in an unexpected way or can be a big component failing spectacularly due to a bug, or race condition, or a multitude of other issues that are not entirely predictable and guarded against at the time of writing such software.
The latter is what has "wear and tear", it's not one software, it's a whole system of software communicating with each other in varying states of hardware decay, you can design and build it to be resilient against a multitude of predictable issues but you can never expect that it will run perfectly fine unattended.
The only good thing is that “turning it off and back on again” works. Usually.
The parent post is asserting that code is inherently stable so long as the platform doesn't change. I disagree.
> we don’t think so. the prod incident we heard about involved someone making an ill-advised choice to reactivate a large account, causing a huge load on the social-graph system, on the night before a prolonged high-traffic event.
Embedded systems that still do their job after 30 years do exist but they live in isolation in a specific and controlled environment, and are built for a limited, unchanging task.
On the other hand, complex web software is build on layer upon layers that are not in Twitter’s complete control.
Hardware change regularly, requiring changes at the lower levels of an OS, inducing potential changes in behaviour, performance, which require adaptation as a consequence.
And that’s before considering security, eternally moving goalposts. Not just at the OS or network level, but also at the business level.
Twitter and al are not living in a locked down context, they live in the messy world of human interactions and that alone requires constant tweaking.
So yes, a binary is more like a mathematical construct and by itself it won’t rot, but if the world around that binary changes, you need to change the binary as well, and for that you need maintenance. The amount required depends on the complexity, brittleness and how well your stack is engineered, but implying it’s a con is a bit extreme.
Sure, those binaries might work the same when executed. Although the probability of that is never 100%, but as you pointed out, the rules of arithmetic aren’t expected to change any time soon. That’s correct. Unfortunately software does not exist in its own micro-verse, it’s subject to the laws of physics acting on the machines it’s running on. So while you might be able to write scripts that work decades later, it’s much harder to ensure those scripts consistently run for decades. RAM chips, CPUs, and everything in between are guaranteed to eventually fail if left running unsupervised in perpetuity. Entropy raises with complexity. At Twitter’s scale, to run a software service you need globally distributed cloud infrastructure. They likely have hundreds of services, deployed to many hardware instances distributed across the globe. Twitter isn’t 1 script running 1 time producing a single result. It’s hundreds if not thousands of systems interacting with one another across many physical machines. Layers of redundancy help, but ultimately cascading failures are a mathematical certainty. Many would argue the best strategy to reduce downtime on these systems is to actually optimize for low recovery time when you do fail.
Software is also bound to the world in other ways. Similarly to how most business processes, products and even more generally, tools, change over time, so too do the requirements placed on software systems made to facilitate or automate these things.
Ultimately the only way to escape the maintenance cost of software is to stop running it. The longer you leave a software system running, the more likely it will eventually stop.
I thoroughly recommend researching entropy as it regards to e.g. information theory, systems engineering and even (perhaps especially) to machine learning.
Computation is ultimately about what we can compute _in this universe_ and the forward flow of time is an emergent property from the universe’s innate entropic guarantees.
Time is “pre-sorted” for us thanks to entropy, enabling us to define algorithmic complexities over the time domain in the first place.
Things last when you take good care of them.
Things do still need to be fixed of course.
Turning a large scale system entirely off and on is never simple. Invariably you’ll run into some kind of circular dependency that must be manually investigated. And even tracking those down becomes tricky.
Classic examples are things like DNS, service locators, or authentication systems. And large tech companies are notorious for NIH-syndrome for all of those.
Twitter doesn’t have unique scale problems by todays standards.
So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.
Did we read the same article, because it seemed clear that all that resilience work was done when the company was public shareholder owned, not under the new private owner.
So operating Twitter with 80% fewer engineers isn't the voyeuristic suicide that many of us are hoping to be amused by.
(after a certain amount of time, or upon default, the debt would convert to shares. but it can be much more complicated, and onerous than that too. lending can be fun.)
It gives them peace
Nobody wants introspection unless it’s to make money on a prediction market
Musk took over and made cuts. I don't think anyone could argue that those cuts were at the cost of Tesla's long-term growth. They were needed because of an imminent liquidity problem. Tesla has hired great people since then.
This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.
https://www.datacenterdynamics.com/en/news/report-elon-musk-...
Now he just needs to shut down the other 2.
https://twitter.com/elonmusk/status/1594865247323660290
Millions of political moderates are stopping by to see what happens when a large-scale public forum begins to tolerate freedom of speech.
The site was also consumed in horrific discourse for the last week (topic being "is making chili for your neighbor racist, ableist and oppressing autistic people?") so all in all it seems to behaving as usual.
That got a Spanish politician (Francisco José Contreras) banned.
No you didn't. You made a bet and got your bluff called.
he did get it right one time though
And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.
A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).
It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail.
Why? What's driving that response?
Was that a real thing those people were saying?
Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
So it wasn't unreasonable to think that Twitter would begin experiencing problems.
I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way.
You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face!If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy.
But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!
But the parent poster characterizes those disliking Elon Musk as suffering from "psychosis" and "desperation."
It's a an unfortunately common, passive-aggressive ad hominem tactic you see on the internet and elsewhere. "People don't like thing or person XYZ? Oh, they must be mentally ill and losers who spend their whole lives obsessing over that thing/person"
HN should be better than that.
Experience.
Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things going wrong.
More so at companies like Twitter where they never really reach a steady state. You constantly have large fluctuations in system stress e.g. World Cup, Trump rejoining etc.
My experience is both exist at the same time because the leadership teams don't actually know what core business is or are busy building empires and resumes.
However it's worth noting that's not what happened at Twitter; there were very specific and explicit reports that the "keep things running" teams were hit just as hard by layoffs/resignations as the other departments.
So there was real justifiable concern there.
There's also a lot of things that can go wrong during sloppy and abrupt handovers. Like... you fired the guy who manages the domain renewals. In the chaos of transition nobody picks this role up. One day 18 months later you realize "twitter.com" has expired. Or whatever. Even if the remaining staff is sufficient to keep things running, there are thousands of these little process interruptions.
The only prediction anyone made was that the World Cup was historically a period of very high load, so if something was going to go wrong soon, the weekend would be the first vulnerable time, and the World Cup finals will be the next.
Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).
e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.
So far, the conspiracy theorists have had a poor track record as far as I can see. That said, I don't engage or try to debate my friends on it, they have always become extremely upset by any debate on such topics. And they're not hurting anybody so my philosophy is just to leave them to their faith. I don't go around challenging people to prove the existence of God whenever they make a religious Facebook post either, I'm not that kind of asshole.
As you can read in the Republican-written Senate Intelligence Committee report, everything about Russiagate did in fact happen and it was 100% real. (except for the Steele dossier/pee tape)
This essentially comes down to Trump realizing that if you say all your crimes in public, nobody will think it's a crime and it won't be reported as one, because you don't "seem like you have something to hide".
edit: All this is relevant, because that dossier itself was the basis for 'Russiagate'. I am ok with a rational counter-argument.
https://en.wikipedia.org/wiki/Steele_dossier#Risk_of_contami...
The conspiracy theory is that Trump colluded or conspired with Putin to hack the election. No evidence to support this has ever been produced. I know that may be difficult to accept, and I ascribe no malice to those who genuinely believe conspiracy theories, they are a powerful psychological trap.
Plus his campaign manager was working for him for free and instead being paid by a GRU agent for their targeting data.
You can try denying the Senate/Mueller reports on this, but you should read them, since they do say this.
This is how conspiracy theories hook people, they tell people what they desperately want to be true, and they falsely claim there is this vast amount of solid, air-tight evidence, and it will all soon come out. That gets people invested, hooked, convinced that it is true. When things eventually get disproven much later and it turns out there is no such evidence forthcoming, the believers are so invested in it that they are incapable of accepting the reality, so they cling to whatever flotsam they possibly can -- "Trump did collude with Putin to hack the election, therefore this joke at a campaign rally must be a message to Putin, therefore it is irrefutable evidence that Trump colluded with Putin to hack the election". It works on probably a subconscious level, it is a very difficult to step back and see this when you are in the hole, let alone dig yourself out of it.
No report into this including the vaunted Mueller Report has ever produced any evidence that Trump colluded or conspired with Putin to hack the election. I have read them. Mueller Report On The Investigation Into Russian Interference In The 2016 Presidential Election, Volume I, p. 181, second paragraph:
For that reason, this Office’s focus in resolving the question of joint criminal liability was on conspiracy as defined in federal law, not the commonly discussed term “collusion.” The Office considered in particular whether contacts between Trump Campaign officials and Russia-linked individuals could trigger liability for the crime of conspiracy—either under statutes that have their own conspiracy language (e.g., 18 U.S.C. §§ 1349, 1951(a)), or under the general conspiracy statute (18 U.S.C. § 371). The investigation did not establish that the contacts described in Volume I, Section IV, supra, amounted to an agreement to commit any substantive violation of federal criminal law—including foreign-influence and campaign-finance laws, both of which are discussed further below. The Office therefore did not charge any individual associated with the Trump Campaign with conspiracy to commit a federal offense arising from Russia contacts, either under a specific statute or under Section 371’s offenses clause.
so the idea we can 'find no evidence' is because people rigged the system so there would be no evidence.
Democrats could start taking money directly from Xi Jingping or from some communist in Venezuela and we would never know about it. Republicans can take money from the Saudis and we would never know about it. This is the reality of modern politics in the US.
infinite invisible money and zero evidence that anything happened, by design
Same as how China secretly paid the liberals to create the global warming hoax, that's another one along the same lines as the Russia collusion one that some people actually believe.
Not to search very far, me and my coworkers discuss politics ( and we do have somewhat divergent views ). The bubble only exists if one does not talk with other people. In other words, other people being like-minded is not an automatic pre-requisite.
But more importantly, why can’t you be friends with people who have different politics? Honestly I’m not sure what word best describes a world where everyone has to think the same to get along, but I have to think it’s far from ideal.
That said, the sheer amount of weird emotion attached to this particular billionaire is just perplexing. Only few years ago he could do no wrong. Today, it seems, he is being ganged up on including by some of his biggest internet fans. And media oblige for whatever reason. I dislike billionaires, but I dislike them as a caste in our societies. I don't see Elon as a savior of mankind or the devil, but those emotions that swirl around him are made to make him look like one or the other.
Somehow, one is supposed to pick sides.
Trump seems to create the same emotions. It is truly fascinating.
New Yorkers despise trump and have for ages. From tenants, to contractors, to business partners they all know what a shitheel he is. Outside of NY though trump was largely seen as an affable caricature of a rich dude. Case in point: NYC as a whole isn't averse to republicans and yet they still rejected trump in 2016.
Elon's the same way. Sure he called that diver a pedophile. But he also had a Tesla employee hounded until he fled the country. Musk had the guy swatted for fucks sake (or tried to at least). Or what about the journalist who pissed him off and ended up having some Tesla features disabled on his car. Or his ex-wife who talked about being treated like an employee. These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight. Now the whole world can see that he went full on Howard Hughes.
This is a very interesting statement to me. I am trying to recall conversations with my wife's side of the family, who supported Trump ( initially, mostly ) and their impression of him was almost exactly that ( plus, "he is a businessman so he knows the game" or something to that effect ). I am in in Chicagoland, but now I am curious if any serious study was done on whether that is how he was perceived as ( ideally before it became socially awkward to state your political affiliation openly ).
<< Their inner circles knew how rotten they both were, but outside of that they were largely unknown.
And that is the other interesting thing. Prior to whatever caused this change of hearts, I only saw impressions of Musk that were similar to impressions of Gates ( you know the ones - saving Africa or something; I know it reads awful, but I really don't remember what his PR presents him as ) - visionary making cameos in odd cultural vignettes ( I think he did one in 'Big Bang Theory' ). The articles describing his work style were very slowly bubbling up.
<< These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight.
And became too much of a power center himself.
I will need to sleep on it, but I think I agree with you.
For those people it has nothing to do with technology, which is why their takes were so hysterical and wrong. It's 100% about politics every day, 24/7.
I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully.
Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter.
But I'm going to assume it's not relevant.
P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of frigging Twitter.
Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?
If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.
If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.
So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.
I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.
However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.
I don't know why this nonsense keeps being spread.
a) Steadily increasing revenue, DAUs etc and benefited immensely from COVID.
b) Were on track to being sustainably profitable.
c) Starting to get traction in stealing ad spend from Facebook who had their own issues after Apple's effects on attribution reporting.
The company was in a much better shape than many other similar SV companies around.
> Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?
http://www.cs.yale.edu/homes/perlis-alan/quotes.html
Anyone who has ever been oncall can intuit how often stuff breaks in big or little ways. Sometimes it's transient and goes away, sometimes it can be filed away to be fixed in the next year, but sometimes, it turns out to be an all-hands-on-deck crisis for a team, or 5.
...for people who understand software to some extent. I get the feeling a lot of people see it more like a hamster wheel, where once the developers are gone it immediately starts noticeably slowing down as it stops (and are confused when that doesn't happen).
Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.
1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left.
2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden.
The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.
No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing.
https://www.cnbctv18.com/business/who-is-funding-elon-musk-i...
https://www.washingtonpost.com/business/elon-musks-twitter-d...
The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR money back with a profit, leave the debt to the company and the sucker who buys it, and look for the next victim/target.
Musk likely went the rapacious route because he made a dumb move, and this is the only way he's getting the money he committed to the deal back.
Oh yeah, and when they buy the company, they are in charge of it. So they can pay themselves whatever they want in executive bonuses / etc.
The people who buy the debt know what they are getting into. (And if those 'suckers' don't know, honestly, they shouldn't be investing in junk bonds etc.)
what if the debt is sliced up and repackaged and rated AAA fraudulently?
https://en.wikipedia.org/wiki/Barbarians_at_the_Gate
Well, yeah. Like the rest of the financial system.
Why not? It's pretty good for all the stakeholders that have a say.
The old shareholders sold for a nice price.
The new ownership didn't have to pay for the whole thing.
The lender gets to charge a pretty good interest rate because there's a good chance of default. If they're lucky, they get repaid; if not, maybe they made enough in interest to make it worthwhile; maybe when it defaults, they'll be able to make something worthwhile out of the wreckage they got at a nicer price.
Leveraged buy outs aren't great for stakeholders that don't have a say. Employees usually get new terms worse than the old ones; in this case, there's been some severance at least. Customers get a promise of a big bang bankruptcy in the near to medium term, rather than a slow fizzle. Sometimes companies with a large payment they can't make can restructure, and sometimes they shutdown with little notice. As a private company without public accounting reports, there will be a lot of guessing about revenue and debt service.
Because at the end of the day, as you mention, it's the employees and their families that will suffer the most. But as long as those employees don't have board seats while strikes and labor-related physical protest movements have become a thing of the past then I guess this is the reality we'll have to live on for the foreseeable future.
In Twitter’s case, its $13B in new debt on the balance sheet means that, every year, they have to come up with $13B times the interest rate in additional revenue and/or reduced cost merely to be in the same place profit/loss-wise. Elon already massively overpaid what Twitter’s business-as-usual can generate even before accounting for that $13B; as a result, the post-acquisition Twitter has to try a bunch of negative-expected-value moonshots in the unlikely hope that one of them hits against the odds and the others don’t turn out fatal, because doing nothing or iterating sustainably kills the company via debt service.
In a lot of cases the best solution would be for the company to declare bankrupcy, reorganize, and discharge the debt by convincing creditors to take pennies on the dollar and a share of the resulting smarter-run healthier company, and a judge that the plan is reasonable. However, Elon both poisoned that well by firing people, angering advertisers, and bumbling around product-wise, and also staked a bunch of $TSLA that would need to be liquidated to go through with the bankruptcy.
Airlines are notoriously for going bankrupt regularly.
"Chapter 7 of Title 11 of the United States Code (Bankruptcy Code) governs the process of liquidation under the bankruptcy laws of the United States, in contrast to Chapters 11 and 13, which govern the process of reorganization of a debtor. Chapter 7 is the most common form of bankruptcy in the United States."
https://en.wikipedia.org/wiki/Chapter_7,_Title_11,_United_St...
Pure copium.
If you don't think he can keep selling 1B a year, I don't think anything I say will convince you.
The idea that the banks will somehow repossess Twitter from an unwilling Elon is detached from reality
In what sense?
1 Toyota Motor Corp 10,495,548 (same) +11.8%
2 Volkswagen Group 8,610,100 (same) -5.5%
3 Renault Nissan Mitsubishi Alliance 7,680,014 (same) -1.3%
4 Hyundai Motor Group 6,667,085 +1 +5.0%
5 Stellantis 6,583,269 +1 +5.2%
6 General Motors 6,291,000 -2 -7.9%
7 Honda 4,121,000 (same) -6.5%
8 Ford Motor Company 3,942,000 (same) -5.9%
9 Suzuki 2,763,000 +1 12.9%
10 BMW 2,521,514 +1 +8.5%
Notice who's not in that list?
Now let's look at market capitalization[2]:
1 TeslaTSLA $568.87 B
2 ToyotaTM $202.86 B
3 PorscheP911.DE $101.44 B
4 BYD002594.SZ $86.39 B
5 VolkswagenVOW3.DE $84.30 B
6 Mercedes-BenzMBG.DE $69.28 B
7 General MotorsGM $57.48 B
8 BMWBMW.DE $57.39 B
9 FordF $56.60 B
10 StellantisSTLA $48.87 B
There is no possible way Tesla is worth more then double what Toyota is, while shipping about 1/10th of the vehicles. The Price-Earnings ratio is 56, vs Toyota's 9.
By every possible metric, it is vastly overvalued compared to it's profitability, and there's no reason on the horizon to think it will suddenly catch up since all those automakers are moving directly into it's space.
[1] https://www.carexpert.com.au/car-news/who-won-the-automotive...
[2] https://companiesmarketcap.com/automakers/largest-automakers...
If you think Toyota's stock (or bonds etc) will increase in market price relative to Tesla, for a fee financial markets will let you bet on that conviction and make money, if you are right.
Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.
Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!
To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.
As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.
Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.
And the house didn't fall down within 3 weeks of it being purchased.
So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…
Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
Said no-one in the entire world except a hand-full of Hollywood studio owners.
That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.
Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
You may not like it, but having a bot do that probably saves a lot of legal hassle.
> (2) No effect on intellectual property law
> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.
If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.
Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.
Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.
It might not be small, but it's not exactly huge.
They already had billing integration, even if you're accessing it via a new route, and they already had a boolean on your account.
Like I said, not small, but not exactly huge either.
Suddenly that simple change had the potential for catastrophic consequences.
I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.
Okay, so two booleans, and checks of those booleans in a couple of algorithms.
This still isn't a huge change, it's not completely new functionality. Once again, not saying it's small, but it definitely isn't huge.
Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.
In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?
When was the last major new feature? The site has always seemed pretty stagnant.
As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.
Whatever. Twitter and Musk deserve each other.
I'm pretty sure there will be no failure at all, and Twitter will work just fine.
Not the end of the site by any means but cracks are showing.
I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.
everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.
Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.
Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.
https://techcrunch.com/2022/11/11/musks-lawyer-tells-twitter...
Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?
Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.
If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.
Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.
If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.
I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.
Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it.
People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems
In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done.
Not quite the same business, but a bit reminiscent.
You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos.
Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.
In modern software environments, the entropy is almost violent -- the changes in all the constituent dependencies are constant and relentless. Something frozen in time does not stand a chance, unless it's entirely stand-alone and dependency-free -- an unlikely scenario with a service of Twitter's size.
The bigger a ship is, the slower it is to turn.
IBM is a "tech" company that employs 282,000 employees, and when was the last time they invented something? I don't remember the last time I heard IBM in the news about something they made.
The bigger the company, you often times find less innovation and more administration & bureaucracy.
The reason startups can survive is because of its small size that makes it very flexible and adaptable to chaos and change, that gives it the edge over bigger companies.
Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart.
The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or elicits help from others on what to say that sounds like tech speak (RPCs!), but when it comes to being truly knowledgeable in this space, he is losing his most valuable assets because of his amazingly poor managerial and ownership style.
I know there are a lot of Elon fans on this site, and will disagree with all of this; but his abilities have not at all been proven. Yes, he knows how to spend money to claim credit for technical advances, but until he actually has his hands dirty in the muck of the hard work of tech, he will always be a glorified self-promoter with no substance.
And Twitter will suffer for it.
"Must" might be.
No one has figured out a way to monetize a social network without advertising, at least not at the scale Twitter has to operate.
I think the statement probably is more like "no one has figured out an easier way to monetize a social network"...
A "de facto public square" would be public in conception, construction, and support from the start, which is one of the ways we know that Twitter is no such thing. Though it would likely also have some rules for how speech is/isn't conducted.
And all things considered, advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse.
This reminds me that progressives have historically always supported corporations as complex hierarchies, scientific enterprises, run (ideally) by “experts.”
But then again, if you're scare-quoting expertise, maybe that's not the conversation you're here to have.
And that's a very progressive sensibility towards corporations, and precisely why they actually like them (what's more appealing to a progressive other than a huge centrally-planned organization run by credentialed experts) despite claiming otherwise.
And corporations are not generally progressive in outlook either, there are too many values higher up on ladder of concerns.
Advertisers are one subset that bring a rough approximation of democratic to their decisions, knowing that each person in the market they hope to reach will be deploying something like a vote with their dollars. As with any democratic approximation, it's only progressive to the extent that population is, though it's poor compared to other democratic mechanisms.
You can take issue with whether democratic decisions are good decisions, of course, but that's likely to be an unpopular opinion for obvious reasons. And hey, I hear the person currently running twitter recently went so far as to say vox populi vox dei. Was his expertise in running corporations part of what you meant to scare quote, or is your challenge to expertise, shall we say, selective?
While most aren't self-conscious enough to realize it now, they still love corporations as evidenced by how they zealously defend them when they serve progressive ambitions and power.
https://press.princeton.edu/books/hardcover/9780691169590/il...
[citation needed]
CNN's ratings were never better than under Trump. He's fantastic for advertising. So is Musk. All controversial figures are. That, oddly, isn't controversial in advertising.
>on a whim!
He created a public poll, and when people voted for Trump to be allowed back won, he unbanned him, tweeting "vox populi, vox dei" ("the will of the people is the will of god"). Had he unbanned him despite the poll saying "no" you could argue it was a whim, but that isn't the reality we're in. He also refused to unban Alex Jones, citing exploitation of child deaths and a personal story. Not unbanning Alex Jones was more whimsical than unbanning Trump was, factually speaking. Why do people always misrepresent his actions? And why is it always upvoted and not flagged here?
It's not about being a fan or not, it's that you're not actually providing any real insight other than signalling how smart you are.
There is an absolute vacuum of technology specialists in the middle of the US, because no one wants to "live in the middle of nowhere," and they don't want to earn less than FAANG (MAMAA?) salaries, when half of those salaries can give you an amazing life in the middle of the country (source: my piss-poor salary compared to yours).
http://paulgraham.com/fh.html
1. https://patents.justia.com/inventor/elon-musk
It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increasingly erratic behavior in the meantime.
Is today's Elon capable of doing what Elon from 15 years ago did at Tesla? I don't think that is necessarily in evidence, much less in a very different industry.
> It's the Halo Effect fallacy to think competence in one field automatically translates to another
I didn't say it was. I was responding the notion that Musk blundered into success at Tesla and SpaceX.
Rolls-Royce (of the last century) would qualify, but it was more aero than space.
Look, I don't know if Elon is a genius or an opportunistic parasite with really good PR. It seems unlikely if we ever will know that. What I object to is people pointing at his ultimate financial success and crediting him with the current result of 2 big companies whose future is very much not determined.
When I look at his process from this ant's perspective, I think he is an abusive unstable individual who takes credit for everyone's work and lies a lot. He also flip-flops depending on the wind. Is that success? Not based on my personal values. Have his companies accomplished a lot? Some of them, absolutely.
The definitions and evidence matter a lot, and I personally don't think any of us are qualified to make blanket statements based on incomplete outcomes. Further, I don't think his other companies that require primarily good engineering are very relevant to inherently people problems, like Twitter. My evaluation of how Musk handles people problems is that he is very bad at them, and I anchor my prediction about his Twitter leadership based on that.
Yes. Social media is easier.
> It's the Halo Effect fallacy to think competence in one field automatically translates to another.
This is precisely about leveraging the Halo Effect fallacy. Elon Musk might not know social media, but the markets don't know that, nor do they care. The average retail trader sees "Elon Musk's company" and buys and holds, regardless of absurd PEs.
Musk knows the power his brand has. He's simply going to use that to pump up Twitter's valuation, all through the virtue of his "halo"
There are no easy answers if you want to satisfy everybody. One easy answer is to stop trying to satisfy everybody.
Tesla, to this day but especially early, had the govt subsidize their products to help make them more competitive.
I don’t think those are even a bad thing, but it isn’t a supportive argument that he’s a great free market capitalist.
This is just a misdirection. I could just as easily said, nobody did it by themselves, including Musk.
What is “it” in this case?
NASA (and the DoD) had vertically landing reusable rockets designed for orbital flights back in the early 1990s. They were being successfully tested but budget cuts killed the program. They weren’t doing “it” because it wasn’t the same priority in that era. NASA has been researching COPVs for decades, etc.
The SpaceAct agreement between NASA and SpaceX allows for sharing of this kind of information. If you think SpaceX has done all their great work alone, you are misinformed and likely making you data fit your conclusion instead of the other way around.
SpaceX has some competitive advantages, but I don’t think they are what you think they are.
Obviously, cheap launches into space.
> If you think SpaceX has done all their great work alone
What I'm saying is SpaceX got it done. No other organization in the world did. The fact that SpaceX had an obvious learning curve of failing and exploding rockets makes it obvious it wasn't just copy and install NASA technology.
NASA's reusable rockets were on the space shuttle, which turned out to be fantastically expensive and impractical.
> NASA has been researching COPVs for decades
Somehow not resulting in a practical, inexpensive reusable rocket.
It's undeniable that NASA has made many great achievements. But making space accessible in an economic manner isn't one of them.
I wasn't referring to the shuttle. Note I said "vertically landing reuseable rockets".
Are you claiming SpaceX doesn't use COPV's, or that don't benefit from prior COPV research? I don't think either position is accurate. Of course no single technological advancement defines space exploration.
Can you elaborate on what you think SpaceX's key advantages are? I can use that to gauge if you really know what you're talking about. I don't want to sound rude, but it's starting to come across as a poorly informed discussion, but one where you have strongly held beliefs. That's not the relationship we should probably hope for.
Ok. So where are they? SpaceX obviously didn't copy a working system, as it took many failures for SpaceX's reusable rockets to work.
> Are you claiming SpaceX doesn't use COPV's, or that don't benefit from prior COPV research?
I was very clear on what SpaceX's success was. Has NASA ever re-used a booster?
> Can you elaborate on what you think SpaceX's key advantages are?
I was very clear on that, too. They provide a cheap way into space, something that NASA has utterly failed on (and every other government, too).
I meant what advantages do they have to facilitate that. You gave me the outcome but haven’t shown any understanding of the why. When somebody asks what makes Tom Brady special, saying “because he wins more” isn’t really saying much and doesn’t take show you know much about football.
I ask because I suspect you will just give some rote public vs. private answer but that’s only a superficial reason. There are underlying systemic reasons, but you need to remove yourself of that false dichotomy first to get there.
FWIW, I’m not a big fan of NASA. I think they are largely a broken culture and a shell of what they were in the 1960s.
I've said it over and over. The profit motive.
I'll try to illustrate a more nuanced perspective. It's no secret that NASA levies a lot of tough requirements. For example, contractors must have a robust pressure systems program. This includes managing/certifying systems all the way down do small air compressors in a vehicle maintenance shop. Same goes for software quality and a million other aspects of spaceflight. Contractors hate these types of requirements because they're expensive. Many within NASA hate them, too. Certainly some of this is bureaucratic overreach, but a lot of it is also good, sound engineering practice. There are mechanisms to waive these requirements, but few people want to openly do so for a variety of reasons. I could go deeper into the why but it's a bit of a digression.
So what does this have to do with SpaceX? CCP, IMO, is a clever work-around to avoid accountability to these requirements. NASA, rather than buying a product, is buying a service. So even though NASA expects them to meet those same requirements, there's very little oversight to force them to do so. Some people have raised flags about these issues but are essentially told to stand down because they don't want to tell the contractors how to provide the service. It also gives NASA a smokescreen to get what they want (faster, cheaper production) while avoiding accountability when things go wrong (they can always point to the requirements they claimed they wanted, but didn't provide the oversight to ensure). NASA knows those requirements rapidly increase costs and on one hand they don't want them, but on the other they want plausible deniability if something goes wrong. Minimizing requirements can streamline the process. The fact that they manage way less requirements is why SpaceX can have a single 23 year-old managing a program that takes a team at NASA. In other respects, it turns a blind eye to the very requirements that manage risk.
E.g., Falcon 9 had supplier quality issues that lost a rocket [1]. Most who work with flight hardware would be surprised to learn SpaceX wasn't applying industry-standard supplier quality checks on critical flight material. Once a mishap happens, NASA gets to swoop in and investigate. And the result is SpaceX now has multiple reliability layers to mitigate that risk. It's not that it was some unknown risk, it's just that they weren't managing it properly. To the uninitiated it looks like they were running a tight, streamlined ship but in practice it was being played a little too fast and loose. Boeing did the same with Starliner but had a bad roll of the dice. The real question is how many times can this happen before SpaceX starts to look like their bloated competitors? NASA could do the same by just peeling back requirements and upping the risk.
Starliner also had it's own host of quality issues as part of CCP. People at NASA were concerned, but their hands were essentially tied until there was a smoking gun in the form of a botched demo that risked crashing into ISS. Again, the "profit motive" at work can sometimes mean more risk than intended.
As a di...
BTW I am a fan of your work.
Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years."
He clearly does know how to make incredible sums of money. Why that's not enough and people need to find excuses to exaggerate or demean his intelligence is beyond me.
Here's two examples I've found particularly insightful that shows he has some ability to talk about engineering details.
This example where he talks about the choice of steel for Starship as opposed to any other metal, something that would be an otherwise unsual choice: https://youtu.be/vLC5W53Fsyg?t=936
This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805
I knew what you were talking about when I started reading your comment. "Make your requirements less dumb" first seems so obvious once you've learned it.
All the denigration directed at him seems to come from people who've only read headlines about him from sources who hate him.
This is the insane thing to me. He's promised a lot of things, but he has also delivered some pretty huge things. Tesla kicked off the electric car migration and has millions of EVs on the road. SpaceX has reusable first stages on their rockets and are the only private company to send humans to space. Just those two things alone are massive achievements. But people look at some things he's promised but has not yet delivered and that somehow is more important than what he has delivered?
Brute forcing a problem with better technology not always the actual solution when technical problems aren't actually the problem.
My guess would be that he has some knowledge but also is very good at faking it which is not necessarily a bad thing - those are good traits for a CEO. Though people should be aware of this fact when evaluating the whole persona.
Actual quote. Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is out of a Dilbert strip.
I've seen a lot of mockery of this request, but I suspect people aren't considering the wide variance in employee quality that can exist within a mismanaged organization. What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.
So it's basically a FizzBuzz test, but for existing employees?
You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It."
That's what's happening here.
That engineer that was humiliated publicly for defending a slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine? Hacker News is filled with people just like him that have built near identical software in other orgs.
Understandably they're upset and are looking for any excuse to dismiss Elon's criticism of not just Twitter, but their entire industry.
Citation needed ;D
If Elon had joined Twitter, and spent time understanding the business and environment and then excised the people he felt weren't contributing towards his vision, that would be one thing... but he has made arbitrary judgements based on absurd metrics like lines of code or willingness to show up at 1am to draw on a whiteboard, he has not made judgements based on the quality of the work or the value people have delivered.
Likewise, to suggest that a software engineer is bad because they were a part of a team that built a "...slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine..." is absurd: what if that person was the only reason that it wasn't 10x slower? What if, they were the lynchpin in that team ensuring that brought everyone else up to a much higher standard which ensured that what they built was usable (even if it was bad)? You cannot judge the contribution of an individual without considering the wider context.
I have no problem with a company cutting most of their software engineers (I encourage clients to minimise their exposure to software engineers, I encourage careful hiring over volume) but what Elon is doing is... not that.
You're probably right, but there's a decent chance that Elon's heavy-handed approach was necessary.
I've seen the "gentle" approach fail.
For example, at $dayjob a bunch of on-prem stuff is being slowly modernised into the cloud. Very slowly. Slow enough to give the dinosaurs time to play politics and protect their turf.
For example, the networks teams that are used to legacy in-line firewalls will cozy up to some non-technical senior manager with a budget and get them to approve a project to roll out this legacy technology in the cloud. That way they don't have to retrain or -- worse -- risk being made redundant.
If instead some team comes in and simply bulk-migrates workloads from on-prem to the cloud... breaking a handful in the process and just fixing forward, then it appears to be messy and crazy, but the effect is that the legacy data centre teams are made redundant virtually overnight. Now they've got no clout, no time, and no pull. They're simply walked, and will find jobs elsewhere.
I've seen both approaches, and the latter style worked better long-term.
Time will tell. I don't think Musk/twitter's case will set any precedent. He is too much of a character to provide broad meaningful insights into industry. Also he has accumulated a list of failures which are rarely mentioned.
It's hard to blame people based on his decades of public behavior and lying about his education, falsely claiming to have a physics degree and to have been admitted to grad school.
In response to someone saying on Twitter how Elon doesn't understand the technical stuff of rocketry, Tom Meuller, former CTO of Propulsion at SpaceX and the designer of many of their engines responded
"I worked for Elon directly for 18 1/2 years, and I can assure you, you are wrong"
https://twitter.com/lrocket/status/1512919230689148929?s=20&...
“She had somehow been able to take and synthesize these pieces of science and engineering and technology in ways that I had never thought of.”
“I never encountered a student like this before of the then thousands of students that I had talked”
“You start to realize you are looking in the eyes of another Bill Gates, or Steve Jobs.”
He also maintained that Holmes was a once-in-a-generation genius, comparing her to Newton, Einstein, Mozart, and Leonardo da Vinci.
Excerpt from: "Bad Blood: Secrets and Lies in a Silicon Valley Startup" by John Carreyrou.
And if you're going to claim that his successes have been due to the people surrounding him who actually know what they are doing, then all that tells me is that you are acknowledging that he knows how to surround himself with people who know what they are doing.
We're not fans (I'm certainly not), but it takes a special kind of mind to look at Musk's track record of successes and conclude that his latest project is doomed.
But I didn't call it "simple", I called it "simpler", and it is.
Which point?
> It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.
While I broadly agree with you, and also agree that twitters founders and/or leaders are better equipped to manage a community than Musk is, I fail to see how having someone with a track record of success (some in turning around failing businesses) automatically dooms the company.
That was the point I was responding to, in the original post. It's also why I ended of with "it takes a special kind of mind to automatically assume something is doomed just because someone with a track record of successes took it over".
For further reading, I'd recommend this article in the Atlantic. The relevant portion is about midway through.
https://www.theatlantic.com/ideas/archive/2022/11/elon-musk-...
And he hates bloated inefficient teams. His decrees on meetings are infamous. Tripling the team at Twitter implies a lot of internal politics, fiefdoms, communication overhead, and generally a lot of headless chickens running around. There's no nice way to fix such a team. A sledge hammer is one way to fix it and obviously he likes getting results quickly.
So, the notion of laying off most of that team was a foregone conclusion. The notion that a lot of the better people would get upset about that and leave as well is also highly predictable. What's left is a team with some gaps but also a lot of breathing room. And he can always lure key people back in by throwing money at them.
Simple plan. It might actually work. At the cost of a bit of drama, temporary instability, and lots of free publicity. Exactly his style. Cringe worthy and effective. I can see the logic here.
The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly manage the community and make the company profitable.
Twitter doesn't have a tech problem, it has a community problem.
Fixing the community starts with rolling back all the things that clearly did not work. He's using the sledge hammer method there too. So, not very subtle but generally just getting of rid of a lot of failed and failing policy.
The technical challenges in speed and scaling are not challenges at all anymore. Twitter built a lot of stuff in house when you couldn't get that stuff as a commodity. That has changed since then. You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
And which ones are those? Knowing what did and did not work is an actual challenge by itself.
> You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
The massive technical challenge is migrating existing infrastructure to something off the shelf, then finding and fixing the new bugs in that existing infrastructure and/or your deployment/configuration. That shouldn't be underestimated.
It's short term disruptive and might involve some fail whales. And then it is solved again. Break it, fix it. Like it or not, that seems to be the plan. If you accept things might temporarily break, it's going to be a lot easier to act.
It's a microblog, not star ship. At least from Elon Musk's point of view, there's going to be a difference between those two and the amount of brain cycles he's going to dedicate to things breaking or not. He's not going to be afraid to break the team (check), the platform (still running fine), or the community (in progress, engagement seems up so far).
You can screenshot this: Musk will cut down costs, make Twitter profitable, and take it public again when markets are better placed.
The markets will give the newly listed Twitter the same "Musk-boost" as Tesla and ramp up the valuation to $100B.
Ever watch someone do CAD/CAM modelling? They need extreme precision of input that AR sausage fingers just aren't going to help with. You need a num-pad and a good mouse with a stepped click wheel.
I get it. We share some similar neurodivergence traits. He wants to be right in the detail. Constantly jumping from interest to interest, seeing the hidden patterns and connections that aren't apparent to others. But there are a times when I know I just need to shut up and let someone more experienced talk despite my brain wanting to lead every discussion right into solution mode, or providing additional context mode.
I've spent the last 6 years in management consulting (without formal business education), I agree with him when he says MBAs are useless. We know that the best solutions come from diverse teams with diverse backgrounds, skills and knowledge. Not 5 clones who know how to build value driver trees, not to say the tools they bring aren't useful, but they can be incredibly limiting.
For someone who hates MBAs he's sure going about this take-over like someone who barely passed one (i.e. knows more than enough to be dangerous). Sure, you're hemorrhaging money in operations. You need to cut costs and find new revenue streams.
What are your biggest costs?
Labor. Slash / Burn. The old McKinsey 7% FTE reduction will give you some extra operating cash from the years remaining budget and you know it's not so much that people (in fear of their jobs) won't just pick up the slack to keep everything moving. Do it quick because you need to rip the band-aid off and get rid of all that accrued leave, restricted cash etc. off your books too.
Equipment. Redundancy? Sounds like unused resources we can fire sale.
Contracts. Renegotiate? The only two meaningful levers are price and quantity. Start cutting quantity now, renegotiate price later.
This is all dummies guide stuff and tends to go terribly in reality when implemented all at once all together.
For instance, research has shown companies that lay-off when under pressure end up underperforming against the ones who chose not to.
Now who's going to help build and operate those new revenue streams?
Quick fixes for a quick buck and a whole lot of extra risk.
What would be a realistic fine in terms of damages (in $$)?
You could argue its actually negative damages.
There have been several front-page posts on HN in the past few weeks about big name company CEOs apologizing for massively over hiring during the pandemic not realizing that it wouldn't pay off once the lockdowns ended and people decided to go back to living their normal lives again.
He's also extremely divorced, keeps having off the record children with his executives, and is having a midlife crisis.
Why do people do this? I mean in any industry. Just why? Is there something I don't understand about executives or C-levels or something?
There may also be some game of thrones shit going on where if a boss and a mini-boss have a baby it solidifies rank and power.
[1] https://www.sec.gov/news/press-release/2018-219
[2] https://www.washingtonpost.com/technology/2022/05/27/elon-se...
I have no doubt he had excellent reasons to buy Twitter, but what he says they are in public probably isn't the whole story.
He uses Twitter non-stop, and has an incredibly curated public image.
- He purchased it because he’s in a right-wing filter bubble and bought into the idea that Twitter is controlled by a secret left-wing cabal and genuinely believes he’s doing the world a favor because he thinks right wing propaganda is actually moderate
- He has the same brain damage as Trump and requires adulation by sycophants and he thinks this is the best way to guarantee his fix and give him control over his critics
- He wanted the brand name and doesn’t care about what services are currently offered under it
- He’s from the southern hemisphere and wants to destroy the social fabric of the United States
- Some permutation of all of the above
Not sure which of these are the truth, but I’m guessing a combination of the first two.
Twitter was poorly managed. It would have been a $100B company in 2020-21 if it had shown any semblance of consistent profitability.
The opportunity here is to:
- Take the company private
- Cut costs by pruning staff and reducing bloat
- Get high-engagement accounts back, which, in turn, will bring back advertisers (perhaps sweeten the deal by promising something with your other ventures)
- Eke out a few quarters of consistent profits
- Go public again, this time with $500M/quarter of profits
- Enjoy the standard "Musk-effect" 40x PE multiple and slow keep cashing out
This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than worsen, this is no longer the case.
There also isn't a measure of employee quality, and layoffs are often not about the individual (e.g. Amazon cutting robotics). I've had no clue how likely I was to be laid off during most points in my career.
I was given a 33% raise, moved to a salaried position, and got a $5000 cash bonus 10 minutes later.
I had mastered the new system we were implementing and converted and supported the remainder of the customers from the legacy system. Shitty support of the product my team of now one actually drove revenue as the customers would pay higher support prices because of the shitty support I offered.
The service I offered was shitty because my job was to meet minimal contractual obligations for multi-decade contracts to persuade the customers to renegotiate the contracts for a product that hadn't been EOL'd. All the while offering onsite conversion services from the legacy product for most of my day job.
What I'm saying is that company is still alive today, they had a pivot point and literally cut 55% of their employees on a single day.
Google wants to beat facebook still, there is one of a billionity pivot points that twitter could take.
It also seems like a lot of the layoffs were achieved by offering everyone the severence of they left.
Assuming only people on visad stayed, I wonder if that skews more or less competent.
The short amount of time they took and methods they reportedly used don't instill much confidence.
There were multiple executives making $10m/yr+
There were board members
There were shareholders
Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I shouldn't. Maybe the effect of all that extra money sloshing around in the system was to incentivize hiring everyone to make sure you didn't accidentally get a false negative, and not all of those hires were good ones.
OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff, so maybe it is just that Tech companies are saturated and have too much capital for their creativity.
I wonder if any science fiction writer could have predicted silliness like Snapchat having fewer outages than serious government websites because Snapchat is better at hiring engineers. Tech hiring is so bizarre.
Our total headcount was under 10 000. Most of that was logistics and call centres (taking retail orders). Our IT dept was literally about 200 ish. All our office staff fit within a few floors of a medium sized building. IT was less than a single floor.
I’d say our software, whilst serving fewer users, was completing tasks orders of magnitude more complex than Twitter. We didn’t have much down time, in fact almost zero beyond physical damage to our networks and the once every few years disaster. Our critical support team was a handful of greybeards who spent much of the time playing solitaire. Code reviews were tight and access to production strictly controlled.
What we did have was a very lean culture, carefully managed over a long time. The company was over 100 years old.
I do not understand how, with far more sophisticated tech, companies like Twitter need so many staff. We could have run Twitter with a thousand folks, including sales. From an engineering standpoint a few hundred would do. And most of us weren’t great engineers. There were a few very experienced freelancers, though. We all worked 0900-1730, but we worked 7 full hours every single day, every minute of those 7 hours. No beanbags. Most of that time was sat quietly coding in a very cleverly designed open plan office (no cubicles) that was like a library.
Looking back, I think the key insight was that we were very cost driven and absolutely kept everything as simple as possible. We had all the usual project overrun problems, usually because things turned out to be more complex than previously imagined. But it baffles me when I hear engineering head count at places like Twitter. If you had given us 1000 engineers, we wouldn’t have known what to do with them.
Cue a team of engineers working for a year+ to build a complicated data pipeline to bring all that data back together into a graph DB (not a clue why a graph DB) and build a DSL to write the code to make these decisions.
That's where the engineers go. It's complexity that is introduced setting off a chain reaction of increased complexity to account for the complexity introduced. Then you need complexity to account for that complexity and so on.
Not that it's all self inflicted wounds. Running a site that counts 1/10 of humanity as active users is a hard thing to do
If you watch interviews with famous developers like John Carmack, they'll mention that working alone scales to about the equivalent of 5x developers. That is, adding 1-3 extra people might slow you down because of the overheads of communication and coordination. It's only around 5+ in a team that there is a definite advantage.
But what are the chances of putting together a team of 5 rockstar developers that all agree on language, style, and vision? Basically zero. So you have to settle for mediocrity. Popular languages, simple approaches, established design patterns.
If you're an experienced "rockstar" developer coding by yourself and use a fancy language like F#, you can outperform a team of 10+ people. If you're replicating a system you've seen already, 20-50 might not be out of the question, especially if you're smart enough to avoid "tarpits" and instead rely on good quality libraries and CotS components like databases, PaaS, and the like.
Organizations tend to bloat.
Random, rapid cuts might not be the fix here, but headcount was too high.
Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up".
Or I tell my boss "code tends to take longer than estimated to deliver".
I wouldn't be given lots of promotions/bonuses with that outlook.
We're talking about a Twitter CEO with $30m/yr+ in total compensation.
There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
We, the people on the outside looking in, have to be missing something.
> There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
Why not? Why would those board members have ever called him out? The more they're paying him, the more they can pay themselves too.
One way to fix this is to make corporate takeovers easier. This sort of cronyism ends up being a con on the shareholders (and the con is especially easy to pull off if the "shareholders" are passive index funds that don't pay much attention to their holdings).
By removing legal protection on corporate raiding, the board+CEO have to worry about activist investors who ask inconvenient questions like "why are you paying yourself $30M a year instead of giving shareholders a dividend?"
https://www.overcomingbias.com/2010/01/enable-raiders.html
Elon himself is a CEO with a better track record than Parag, why is he not more legitimate if we are pretending leaders can't make mistakes
There's a clear correlation between org size and CEO pay, even when not in the company's interests. Some CEOs and managers also love the power. Politics is also at work.
I think you’re confusing takeovers with mergers/acquisitions
Jack, who was once the CEO of Twitter, has said that they hired too many people too fast.
Elon, who is "paid" way more than $30 million/year to run several companies, also apparently thinks it was very bloated.
I'm not necessarily saying the billionaires are right and the millions are wrong. I'm just pointing out the clear counterexamples to your appeal to authority. I honestly don't understand why you would be so surprised to find out that millionaires also make plenty of mistakes. The Twitter stock price to a degree is one indicator that the Twitter executives were making mistakes.
For the same reason that colleges and universities have seen their administrative bloat skyrocket at 10x the rate of student enrollment. Administrative bloat inevitably creeps into all large organizations. Many of the people in the trenches making hiring decisions weren't considering the overall financial performance of Twitter as a company. They were making hiring decisions based on what was happening in their own department, or how that decision would help advance their own agenda, or increase their budget, or increase manpower on a favored project. When you further consider that many at Twitter openly conceded (and in many cases, bragged about) that they viewed their role at Twitter as moral arbiters of society, crucial to policing the discourse of the public, it is not hard to see how enlisting as many true believers as possible to the cause would be seen as desirable, regardless of the larger financial implications.
And the leads installed by the acquiring company seem to be highly intelligent - the company continues helicoptering in their brightest minds to fix this mess. It just so happens they are forced out every 12-24 months because it’s almost impossible to deliver anything there.
For example, last year they introduced SAFe. That caused an exodus which basically equaled a brain drain (still ongoing). Most people I valued for their straightforwardness and ability to cut through BS there have left or are leaving. The person pushing for SAFe introduction has resigned already.
The founders of that company have all left ofc, and are basically trying to restart the same company (or variations) with their own money. And their products are already picking up speed, becoming threats to the original one.
Come on, just look at the tech industry. When rates were low and stock prices kept going up, "headcount" was used as an indicator of future growth. Grow headcount, investors are happy. After all, the promise of tech stocks was "growth". Usually you're not looking to cut costs until you think growth is over. Of course, Twitter was a dog and did nothing useful for years, no innovation, no new products, nothing. But tech investors definitely saw rising headcount as a good thing...
> These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
Wealth is not a valid indicator of ability.
I'm not judging the execs and board members individually but rather questioning your assumption. I have read you mention "supposedly", yet it can be read as a rhetorical term.
Hiring often isn't done because of current requirements. Senior execs come and go and with them so do strategic objectives. You accumulate people and they're often not laid off when the thing they work on becomes redundant. Large scale layoffs are awful for morale and usually only come after a 'crisis' occurs.
Now the systems are stable but human workers either be sick, leave, or die eventually.
Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for the human workers unexpectedly not working anymore. Have a multiple workers doing the same things improve stability.
Sure, it's inefficient in terms of money. But alternative is one sick important employee catch a COVID-19 and die lost the knowledge of the system. Documents doesn't solve it because you want the manual operation available right now rather than a few months later when replaced workers learned from the documents.
People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…
Well, yes, but also that's super reductive.
The primary reason people work is literally for money. The specific _place_ people work is for things that interest them, engage them, challenge them, provide a path for continued growth in their career, or contribute to a common good they believe in.
And I would argue that it's not necessarily a given that people will be "more engaged and more excited about their work if they were paid more," but instead, may be willing to stick around in an unengaging, unexciting environment _longer_ when they're paid more.
Theories like Hertzberg exist for a reason and show that money isn't even close to the most important thing for people to work for. If the money is good enough, there are many other factors that are way more important. Not enough money is a reason for people to leave, getting more money is hardly ever a reason to work harder. The possibility of getting more money if they work harder is a good motivator though, once they have it that stops.
In most software engineering jobs, people have enough money to live an enjoyable life. At that point, having more money is not really such a strong motivator.
https://www.tiktok.com/@realpankhilpatel/video/7159187292631...
Normal people don't have vacations like that.
Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. As always, the last 2% is a hard problem and business doesn't really want to deal with a such unreliable platform. You wanna onboard big advertisers which potentially spend $100M ARR? Then you need to assign a dedicated account manager to handle all customer escalations. PMs then triage and plan their feature requests and later engineers implement it. Which all adds up.
And they also uses your competitor's product, like Google, FB, TikTok etc etc... Twitter is a severely underdog here, so you need to support at least a minimal, essential subset of features in those products to convince them to spend their money on Twitter. That alone takes hundreds of engineers, data scientists and PM thanks to modern ad serving stacks with massive complexity.
Yeah, it ultimately boils down into a simple fact that it's really hard to take other folk's money. You need to first earn trust from them. They want to see if your product is capable of following a modern standard of digital ad serving for now and foreseeable futures. Twitter has spent lots of time for earning trusts and the original post is one evidence of such efforts. And this usually needs more man power. You might be able to do that in a more efficient manner, but I don't think that's as simple as firing 75% of your entire headcount.
This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outages are short and far between, so they never actually cause real issues. This is obviously costing Meta/Facebook a lot of money, but allows them to be an essential service. So essential in fact, that every major news outlet in my country sends a push message as soon as Whatsapp is down.
If Twitter wants to be a comparably important platform, they need that same stability. And Twitter, for me, is very much the best place to stay up-to-date on any current event (in near real-time). Reddit used to be pretty good with Live, but that's pretty much died (and was mostly a summary of tweets anyway). I really hope Twitter survives Elon, because I don't know of an alternative right now that has the same value in this use case.
What Twitter is doing is to scale down first, focus on the product, and once it gains traction, it definitely can scale up again. I don't think it will hurt the product very much.
I don't think you have a good understanding on how those companies are growing and scaling out. Don't take growth for the granted. "Right product" or "Right technology" won't give you that. It only comes from solving thousands of very specific, never-ending customer problems. If you do B2B, you need to spend most of your time on very specific requests from priority customers. And they are not one, but hundreds of them if you targets $xB business. It's just physically impossible to keep up with a small team even with a very aggressive prioritization.
Still not convinced? Google has a notoriously bad reputation for their customer supports and it's primarily because of their tendency of keeping "inessential headcounts" low as possible. And think about how many cloud customers they lost to AWS and Azure. TK came to GCP and his first work was adding an army of sales and account managers. This almost immediately yielded a rapid acceleration of the platform, although it's too late to catch up.
The freeze worked: we got rid of some products that weren't getting traction and were able to improve the products that did have traction. But the cost of the freeze lingered for at least a year; it reset the hiring pipeline, we couldn't grow fast when we needed to because the limited number of recruiters we had were already overworked, and the limited number of engineers had to balance interviewing needs with their real work. This all happened when my employer was <10% of its current size and pre-IPO, and we didn't even take a headcount reduction.
Twitter is simply at a different scale. 7500 -> 2500 employees is a 66% reduction. Going 2500 -> 7500 is a 200% increase. Recruiting is likely totally gutted, and the current 2500 employees have to support systems previously maintained by a 7500-person company. If they decide they need to grow, it'll have to restart at a snail's pace, and they'll have to make sacrifices on feature development or stability along the way.
Edit: for what it's worth, the fastest way to regrow back 200% is to rehire the people laid off. But, given that I happened to interview earlier today an ex-Twitter candidate who didn't make it through the Elon snap, that route is rapidly closing up.
I mean this is just wrong. Companies are always under pressure to cut costs (employees) and it is always talked about when quarterly results are posted. Look at how the market reacted to Facebook's latest results and then again what happened when they laid off thousands of staff.
This is not a trivial task. With such a heavy reduction and how entire teams have been completely decimated, there will be a lot of lost knowledge. I'm sure there will even be cases where the people who stay don't even know what knowledge was lost.
All that puts Twitter in a very risky position, specially in a product of such complexity that does a lot of things in-house. It shouldn't be underestimated.
Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had < 100 software engineers. A few short years later and they were re-using rockets, a feat many people had thought unlikely/impossible and requires some hardcore software eng.
Not surprised Elon Musk thinks he can run twitter with a skeleton crew.
2) WhatsApp does not support the type of public broadcasts done at Twitter, and due to its e2ee doesn't require much human moderation.
2. WhatsApp didn't have e2ee back then. The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
Your point 2 sounds like there were additional factors that could have influence the reliability besides the load (they didn't simply migrate to FB infra but also switched to Signal).
> The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
There's push-notifying millions of devices within seconds after a celeb or a major news source tweets. There's tracking view and engagement stats on that in realtime. There's making sure a tweet is not available to any of those within seconds after it's been deleted or moderator. There're separate back-office apps for moderating that firehose of content. And that's just what I can see from the outside. An e2ee instant messenger with size-limited chat groups doesn't even come close.
Please don't say "just stick a CDN on top of it and you are 99% there", it's embarrassing (and not to twitter). This will maybe get you 80% there if your goal is "a microblogging platform" but not even 20% if your goal is being both the go-to news source and shitpost forum for people worldwide reliably working even in sensitive times and emergencies. Twitter used to be a microblogging platform back when it had much fewer employees and you'd see a fail whale regularly even as it had much fewer active users, in recent yeas it's a completely different beast and saying increased headcount is unrelated is amusing.
The push notification argument is also overstated. Sharding and fan-out solves the burstiness. And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter. Besides, these days the push notifications go through Google/Apple servers anyways to reduce the number of open connections needed on the phone side.
Then there are DMs. They are per person so CDNs don't help much (just static assets), but also they shard basically perfectly. So, shard them.
Which in the end leaves the user feeds. Designed correctly, sharding would work extremely well, and what doesn't work could be handled by caching closer to users for those 1k most popular accounts.
Honestly, with the correct architecture, languages and tooling, it could be handled by an experienced 50 person dev team plus another hundred in ops. Obviously Twitter doesn't have the perfect setup, so maybe an order of magnitude more? And if you throw a bunch of subpar engineers and tooling at the problem, nothing can dig you out of inefficiencies at this scale anyways.
And no, I'm not wildly optimistic here. StackOverflow still runs off of 9 on-prem servers [0]. I've seen message queues that can give 200M notifications per second on a single machine (written in C++, for HFT). This stuff is hard yes, but throwing more bodies at it doesn't help past the point your fundamentals are solved.
0. https://www.datacenterdynamics.com/en/news/stack-overflow-st...
They switched to a new protocol and grew from 200 million to I guess about a billion users since 2013. If you believe a team of 50 developers could deal with this and not cause extensive downtime and service disruption along the way I pray you never ever manage software engineers.
> Sharding and fan-out solves the burstiness.
Great, at least it's no longer "just add CDN to solve 99%" here;)
> And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter.
Yeah again WhatsApp has many users but as an engineer you just don't ever have to worry about delivering a message instantly to more than 32 people (512 as of this year), and you never have to account for moderating any of that because it's e2ee and there are no adverts next to the messages. It's basically dumb pipes terminated by one native client. Twitter has to maintain a mix of automated and human review of all UGC and is accessible via extensive APIs and search engine indexed web app in addition to native client.
> Then there are DMs
Let's ignore Twitter's DMs, even without them it's far more complex and demanding than an IM app.
> StackOverflow still runs off of 9 on-prem servers [0].
Yeah, and SO maintenance page or read-only mode is up about once a month and lasts dozens of minutes. What are you even talking about now bringing up a niche programmer-oriented help forum for comparison here?
You may be stuck in the times where Twitter was a RoR-based microblogging platform. It's not been that for years.
This isn't 2005 anymore, we have multiple parallel 40gb LAN, 64 cores per socket and 2MB of L2(!!!) cache per core, and a full terabyte of RAM (!!!) per server. If you program with anything that makes cache-aware data structures and can avoid pointer chasing, your throughput will be astounding and latency will be sub-millisecond. How else do you think WhatsApp managed 300M clients connected per server without having just the in-flight messages overflowing memory, on top of all the TCP connection state?
Things only get slow when scripting languages, serialisation, network calls and neural networks get involved. (AKA "I don't care if you want docker, a function call is 10000x faster than getting a response over gRPC and putting that in the hot loop will increase our hardware requirements by 20x.")
The more distributed your architecture the more network overhead you introduce and the more machines you need. Running the WhatsApp way with less, higher performance servers simply scales better. Just from the hardware improvements since 2013 there was no reason for WhatsApp to change their architecture as they grew.
And if you think rolling out a new protocol while maintaining backwards compatibility is hard and somehow adding more people will help, I have a team of engineers from Accenture to sell you. I did this straight out of university, to thousands of remote devices, over 2G networks, with many of the devices being offline for months in between connections. You just need a solid architecture, competent people and (I can't stress this enough) excellent testing, both automated and manual. And the team that did this was 6 engineers, and this wasn't their only responsibility.
2) Public broadcasts make a lot of things easier because that means more of your workload is relatively straightforward caching (as evidenced by this blog post).
So, you don't have the numbers.
> easier
Have you even thought about moderation and all the other concerns that go with this? How does instantly notifying millions of devices helped with caching, for example?
https://nitter.lacontrevoie.fr/libsoftiktok/status/158539526...
It's largely focussed on the event stream behind the core service and data analytics. There's maybe one entry on the main data store and one on search over the last few years.
They had wine on tap.
All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point.
Maybe it was close (in reference to external factors not implied in your very insightful post!) but its amazing how confident people are with opinions on events they have 0 insight into. Everyone knows how to solve the war in Ukraine or world hunger but how rare is the "consensus" (in terms of up-votes or popularity) right... just something that got me thinking. Thanks again for this article. I always love seeing details of tech ops!
Sure, it’s plausible they were made more likely by so many people leaving, but they’re not exactly the meltdown people predicted.
Our expectations were anchored by the idea that Twitter would cease to exist. So very real problems like 2FA going down look trivial compared to the former.
That’s funny but at the same time it’s easy to find stuff like that on youtube and has been forever.
It's still entirely believable the site will go down and remain down for a week or two at any time.
Neither of those extreme viewpoints reflected reality accurately. There were significant problems, but they did not take the entire site down or anything. In fact, for millions who did not have 2FA or use it at that point, they did not see any issue. Whereas people that wanted to find an issue could go looking for the movies. So each side was able to find ways to reinforce their alternate realities.
Personally, I doubt that it will be anything technological that ends Twitter. It will be economic. Their advertising revenue has been decimated and their operating costs have never been higher.
Was there more I missed?
My theory is that the economic problems have been stressing the systems and that makes the "bones" more apparent. The core concept of this country is checks and balances of two opposing sides. I think it was a great improvement over authoritarianism, but you can see that it is definitely not the ideal final paradigm.
1) The site goes down [you look like you can't do you job]
2) The site stays up [you can do your job] but the lay off looks like a good decision.
You are screwed either way.
Some of the side effects of that:
- Trends are not working correctly.
- Copyright reporting is not working.
- Appealing flagged tweets isn't being responded to.
It's only a matter of time before these get abused with no one to fix them.
"Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now?
>- Appealing flagged tweets isn't being responded to.
Mine was responded to in a handful of hours. Much faster than I was expecting.
>It's only a matter of time before these get abused with no one to fix them.
"the walls are closing in" - https://www.youtube.com/watch?v=rLEchPZm318
The team would moderate junk trends. So some of the trends are literally just random words being spouted. Similar to early day twitter.
> Mine was responded to in a handful of hours.
Recently? That's impressive when the related teams are gone.
> "the walls are closing in"
I am not sure what a YT video about Trump has to do with my comment.
I refer you "in comments": https://news.ycombinator.com/newsguidelines.html
moves like musk's are bad for the industry. they produce more systems with "job security" built in.
How common is this?
At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job.
The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass.
Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum.
America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired.
Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.
Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment.
Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations.
The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.
So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough.
It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.
Somehow, some folks are now saying this is bad.
Stop worshipping billionaires.
But he did change the world with his companies already, didn't he?
So at 5% interest the yearly cost is 650 million, even at some obscene and unrealistic payroll per employee (including not only developers), like let's say 400k he'd have to fire over 1600 people just to balance this out. There is no way to spin this positively. Also due to no more RSUs cash comp for remaining employee would need to increase.
Add to that the the loss in advertisement revenue do to obvious reasons.
Taking these facts into account could you please explain how is Musk NOT a liability to Twitter? He basically spent 31 billion (plus 13 billion he offload to Twitter) just to buy the trademark and the user base (and some proportion of advertisers...). From a financial standpoint that's an objectively terrible deal. Musk might have other goals and that's great but in no way this is comparable to Tesla or SpaceX.
>But he did change the world with his companies already, didn't he?
History is full of examples of extraordinary successful people who get blinded by their own success and start thinking that they are infallible. This ussually results which in pretty unhinged behavior when they get older. Musk does not seem like the most grounded person ever so he's likely at a much higher risk of this than almost every tech billionaire I can think of.
Maybe somebody with Musk track record can be seen as more credible than some random "qwytw" on the internet? I mean incentives are on his side, aren't they?
For one his famous. It's his toy, so he won't let it disappear into irrelevance without fighting for it. There are not many people currently alive with comparable track record, being recognized world wide, with deep pockets like him.
What if in few years time from now people will be stepping on each other writing posts and books on how they really knew all of that from the beginning, that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
To be honest, I don't even think anything extraordinary needs to happen for all of it to work out just fine.
For all people saying technically it's not just rendering short messages - well it is just that at the end.
Look at stackoverflow - most people don't realize it's just single monolith application running on 6 servers - all of it.
There are still thousands of people employed there, which feels like too many if anything.
Did I misrepresent any facts or figures? Did I lie? What does credibility have anything to do with anything otherwise? It's an objectively bad deal from a financial perspective, if you have any arguments against that you're free to share them.
Twitter might do "fine" as a platform. Sure, why not... Does not change the fact that he grossly overpaid and that the 13 billion in additional debt will be financially crippling for the company. He might sell some more Tesla stock to offset that. I mean it's his money, he can do whatever he wants (unless the SEC stops being a complete joke).
> that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
Right about what? Even Zuckerberg seems to have a clearer vision about the Metaverse than Musk seems to have about the future of Twitter. I might be wrong. Care to enlighten me instead of talking about some random tangentially related things?
> There are still thousands of people employed there, which feels like too many if anything.
Sure, can't really disagree. However is it better for a business to have an additional 2000 employees or so who are possibly superfluous or to fire all of them and spend ~600 million or thereabouts servicing a debt incurred for no reason? It's a pretty straightforward you chose to ignore for some reason.
Musk has successfully run companies in the past and is running two extremely successful companies right now.
In case it isn't clear, I agree with the parent comment. In the same tone as the comment above the parent, you could also write "Thing is - Elon Musk knows a thing or two about sexual harassment lawsuits. It's inevitable that he will have another at Twitter."
Stop idolizing billionaires.
>Musk has successfully run companies in the past and is running two extremely successful companies right now.
Is he actually running those companies, or is he posting selfies of whiteboards at 1:30am showing architecture diagrams of Twitter?
In these markets, after he was nearly forced by courts into completing a purchase deal for Twitter after making an overpriced offer, I'm genuinely curious to see how he pays the ~$1b in interest per year on the debt to purchase Twitter.
Stop saying this in discussions about one particular person.
It’s as useful as saying “stop idolizing millionaires” when someone says they like Hillary Clinton as a Presidential Candidate.
People who obviously know absolutely fucking nothing about coding or manufacturing or running businesses or software engineering or cars or space or rockets or emerald mines are continuously defending Musk, and their only justification is that "he's rich, so he's obviously smarter than you!"
No they are not. They are idolizing him specifically because of the success of Tesla, SpaceX, and PayPal.
> "he's rich, so he's obviously smarter than you!"
Nobody is making that argument. Its usually, “look at the success of the companies he leads”.
You can make an argument about whether or not he should get the credit he does for the success, but nobody is idolizing the wealth.
None of these people are going to defend the great insight of the winner of the last powerball because there is no repeated pattern of long term success.
This is a vapid over-generalization designed to dismiss any statements by attaching them to the wealth, not the person. Don’t do this.
Nobody is talking about the Waltons, Buffet, Larry Ellison, etc. This is a discussion about Musk specifically and his history of operating companies.
The billionaire worship is simply the secular version of this faith. X made billions of dollars by owning a business that does Y, therefore X must be an expert on Y, and businesses in general, and is a genius, and is a morally superior person. Otherwise X wouldn't be a billionaire, because - horror of all horrors - the world is just, isn't it? It couldn't be the case that X just got lucky, can it? Surely, X has earned their billions, right?
People are “worshipping” Musk because of his repeated success in making successful large innovative businesses. The wealth he has is a product of that, but the deference is to the success of the companies.
[Narrator voice] He did not.
Winning government contracts over other competitors is not the same thing as “subsidized”. Otherwise Hilton is subsidized because government employees stay there sometimes.
To be clear, I'm not implying that's a bad thing. These companies seem like an example of subsidies working as intended, i.e. driving innovation.
In January 2020, Toyota passed the milestone of more than 15 million self-charging hybrid vehicles
If every vehicle sold by Tesla since its inception up to the most recent Tesla sales figures released for Q1 of 2022 are accounted for, there are over 2,645,000 Teslas on the road worldwide
Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.
Another company he runs, Tesla, is famous for being one of the few car companies that makes software for their cars that people actually want to use.
If a citation is needed, it is a citation to explain how anyone could possibly believe with that track record that Elon Musk doesn't know how software is built.
Sure he might not be a whiz with React like me or the OC, but... ¯\_(ツ)_/¯
With regard to him firing too many people, see my thoughts about what his mindset might be here[1].
[1] https://news.ycombinator.com/item?id=33703280
But don't you think it's strange that (assuming so much of SpaceX's success is owed to Shotwell) nothing like what SpaceX has achieved was achieved at her previous gigs? But then she goes to work for Elon Musk and suddenly...
Regardless, the point I'm making is not "Elon Musk wrote all the software at his various companies himself" but rather "Elon Musk founding / running / exiting a variety of companies in very diverse fields but all of which are good at software is a very strong indicator that he understands software and how it is built".
Meanwhile, the "he doesn't understand software" corpus of evidence basically amounts to "I think he's doing a bad job of running Twitter in the handful of weeks he has been there. I think this even though I have very limited insight into what is actually happening at Twitter, besides what is being fed to the media by disgruntled employees who probably hated Elon Musk before the acquisition and their subsequent firing".
So yeah, in my opinion the pro-argument wins.
Not to be dismissive of landing an orbital rocket, but that's mostly a feat of aerospace engineering and control theory. On the software engineering side of things for SpaceX I think their work around CFD would be the more impressive feat from a computer science perspective.
Elon just made sweeping changes that cratered their ad revenue and took an axe to engineering teams all over the company. He did this in the lead up to the holiday season which has been the most profitable part of the year with advertisers pouring a significant portion of their yearly ad budgets into ad buys. Even if he was 100% correct that all of this was bloat, it was a monumentally stupid idea to make these changes now of all times. I would have expected him to at least have enough of an understanding of software engineering to know that deprecating systems and cutting out bloat needs to happen carefully with the talent onhand to be able to back out changes and manage the transition seamlessly. Fundamentally the risks involved are more financially impactful than the price of keeping headcount around for another year to do it safely.
Twitter hasn't even had a major incident yet and just the revenue loss from advertisers pausing ad campaigns probably already offsets all the savings they were hoping for in 4 - 6 months once the layoffs finally start bringing down staff costs.
He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random.
1. https://mobile.twitter.com/elonmusk/status/15921762028730859...
a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time.
b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
c) Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
Elon accidentally conflated the network path of (b) with the performance issues caused by (c).
They're both bad! They both need fixing! The fact that Musk mixed up doesn't matter.
Isn't this not true though? Based on the responses to his Tweet only one GraphQL request is made which batches the whole thing.
> Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
But what does this have to do with India? If the slowness is due to calls between microservices hosted on (from my understanding) their own hardware, why would it matter if the initial GraphQL request comes in from India, since all the rest of the backend requests would be the same?
Literally bang in the middle of an F12 network trace showing dozens of requests, mostly JavaScript. Not JSON RPC, per se, but still an overhead taking up seconds, especially on low-end Android devices.
> But what does this have to do with India?
Everything. Nitter uses 40KB of code & data to display the same content that takes Twitter megabytes.
People in subsequent threads have pointed to the gigabytes of "data" usage by the Twitter Android app, which is just absurd for an app that displays primarily text.
It's a bloated pig of an app that performs poorly on 5G networks in the United States, and hence terribly on slow and overloaded networks separated from Twitter's data centres by an ocean.
"It works fine for me" says the engineer connected to the WiFi in Twitter HQ.
I'm pretty sure that's my point? How are you equating "dozens" to "1200" (or even "hundreds" in your last comment) as though Elon is in the right ballpark?
As for the rest, I don't know what your point is, where did Elon say anything in that thread about how much data they were sending? He's quite literally responding to someone saying it was the amount of JavaScript being downloaded and telling them they're wrong, it's the "1200 RPC calls". I'm pretty sure the point he was attempting to make is that it's slow due to thousands of serial "RPC" calls made between the app and Twitter (which will have a higher RRT to India) but as you've demonstrated that's not what it's doing.
I'm not saying Twitter is well designed, I'm just not pretending like Elon somehow had the right idea when you've had to stretch it this far. If he had a proper technical understanding then we wouldn't have to have such a discussion of what he "actually meant".
This is not true at all. I have a look and it's not that there are many REST requests it's that the requests send back a load of data. For Example the timeline sends back a 1MB JSON response. It's heavily gzipped so comes in at 100KB over the network but it takes quite a while for the whole request.
In places like india the internet is slow and many people are using VPNs. I emulated this and I can see that over VPN on a slow internet connection it's like 10 seconds to load the home page.
But he conflated two things that he didn't really understand and reached a completely wrong conclusion. He's shared that conclusion with confidence and never admitted he was wrong. That's a fatal flaw for someone who holds a Chief Engineer title at a rocket company.
He's a CEO who is has been incredibly successful at selling a vision and getting people to do what he wants. He is not an engineer. He should stop role playing as one.
Its a matter of approach - if you focus on all his success, you can continue worshiping him as some capitalist messiah. If you focus on bad stuff, he is a fucked up mess to be polite, a bad parent and husband, an even worse human being in general, utterly horrible boss, person with deep mental issues that aren't clearly tackled in appropriate ways.
The truth, somewhere in the middle of those 2 extremes, still ain't that nice IMHO. He burned basically all good PR capital be built over the years, basically nobody sees him now as good person, just another ultra rich a-hole like Bezos that needs to be kept in check. As he will find in following years, people's opinions trickle down to cash flow. If I would be buying an electric car right now (which doesn't make much sense for another decade to me), Tesla would be off the table for example.
Tesla and SpaceX have been wildly successful and he will forever recieve some well deserved credit for their successes but at the same time I think its pretty bad that Tesla (and SpaceX) have had multiple near death experiences and some them could have been avoided if it weren't for incompetence.
I followed the Model 3 production religiously and the amount of dumb mistakes they made even after other industry experts were yelling at them about imminent iceberg collision is just unbelievable. In fact the Model 3 production, with its horribly designed body that he wasn't keeping on top of and Musk's stupid attempts to automate all of assembly should be entirely attributed to him. It was so bad it was worse than even the shortseller's worst predictions.
I guess you can give him a pass for near bankruptcy #1(Roadster roll out) and #4(COVID lockdowns). Not sure about #2(Model S early sales woes and potential acquisition by Google)
All I am saying is that he fucks up A LOT and has thus far managed to stave off the consequences of his decisions but many occurrences have been down to luck and luck is something you should never bet on.
Also new owner: "What even is Mesos? Why are we running something called Aurora? Obviously pure bloat. Fire the lot of them."
I'm not a friend of Elon's, but outside of the flashiness of the whole thing, I don't think his firing spree was wholly unwarranted.
The other day I saw a video of a bunch of people at twitter leaving that have been there for a decade or so. I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
This is honestly uncalled for. Job hopping every 2-3yrs should not be an expected task.
Changing jobs is a skill/knowledge net-negative for the employer, and can be negative (outside of salary) for the employee as they have to relearn everything (relationships, tech debt, processes) etc.
Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product".
I'm not that optimistic about Twitter long term - never was TBH but this Musk thing is turning to a shitshow and also exposes what it's like working for him in his other companies.
I'm staying and I'm glad the teams are trimmed.
The excuses for bloat here is just mind-boggling, unless reflects that many people on HN are bloaters themselves
What is a "legal assault" and why does Lichtenstein have an ongoing "legal assault" against twitter?
I've had coworkers hounded by managers from unrelated teams because my coworker wasn't working fast enough on a feature that was important to their team. They made assumptions about how many different tasks my coworker might have had assigned to them.
They were juggling like 3 or 4 fairly important features/fixes for various parts of the organization at the time because our team was (in my opinion) understaffed.
But of course, each of the relevant managers can only see "didn't satisfactorily complete my fix in the expected timeframe", and then perf review comes back and collates that info to "doesn't complete tasks on schedule" without taking a step back and recognizing that this is a failure mode of planning where one engineer was responsible for way too much stuff with no reasonable ability to push back.
At the same time, of course, engineers were getting promotions for doing exactly one thing well, which is as it should be.
There are lots of lessons in here, one of which is very likely "push back on work that causes you to overburden yourself", but I'd argue another important lesson is that you just don't see what others are working on. I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
But while we're throwing ad hominems for the fun of it; I'd rather have a bunch of "bloaters" on my team than the one person who is convinced that they are one of the golden few producing value, and everyone else is a lazy freeloader. I've worked with that type of person before and it's awful. It is rarely the case that they are _actually_ producing an 2-3x the value of other engineers, it's much more likely that they're just reducing morale and causing internal conflict while building some small piece of the pie and assuming that's the majority of the work.
Like someone who builds a button that says "delete my account" and says "I built the delete account feature" while not recognizing that there are a ton of people on the database teams that made the feature possible without performance hiccups or leaving dangling foreign keys.
> I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
This is not the case at all in many work situations, a prime example being when you work on a team and all code and work produced is shared. None works on other teams. The code reviewer and hopefully project lead will know, or have a very good hunch, what everyone is producing.
I would argue that bad middle management is another bloat problem, so in essence; you just made another point for how bloat is very bad for an organisation. I wonder why you are so interested in defending the all too common effect of corporate bloat in an economy where its calculated that a large percentage of jobs are useless and there is a trend to collect multiple useless jobs.
Absolutely. I also realize that when a company reaches the firing squad stage it's not a good idea to stick around - better polish up the CV and move to greener pastures. I have no insider information - they could be offering generous (and credible) retention bonuses for critical staff, but I've seen this happen twice before indirectly (I was gone before it started happening) and the people that stayed didn't have a good time, were lied to and left in the end anyway in a worse situation (or waited trough to bankruptcy and had to sue for outstanding wages in first place I worked in)
What is wrong with that?
Very few industries require constant learning for the business to compete so a highly tenured employee likely hasn’t learned anything new beyond minute process changes for 10+ years. Once people are ossified into a role like that, they will do anything and everything to shut down anything that has a whiff of threatening their current role.
Some were resisting change. Some were embracing it. All in all, I had a great time and gained a lot of respect. I love working in teams with all ages.
Your sentence "once people are ossified..." feels like a generalization to me.
On the other hand the slash and burn Elon approach seems objectively terrible. Indiscriminately firing most of the company kills morale and is likely to send the company into a hiring death spiral where your good employees leave and you can't attract good talent. This won't automatically kill the product or the company but it's not going to lend itself to big positive successes in the future.
German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can speed up your work: whom to ask on the "kurzer Dienstweg" aka short-circuitting bureaucracy when needed, personal relationships with people in other departments on whose knowledge you rely (it's one thing if you get a random email asking for some shit from someone you don't know, but I'll always find some minutes to help out someone who has helped me out in the past), all the domain-specific knowledge about the precise needs and desires of your customers...
Attrition is bad for a company as a whole, the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact), and so there is no KPI for leadership other than attrition rate itself.
The only problem is that over the last years, employers' mindset has shifted from regular wage raises to paying the bare minimum which makes changing jobs every few years a virtual requirement for employees to get raises, and so we are already seeing the first glimpses of US employment culture and its issues cropping up.
This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in companies like that.
This was basically Google’s entire pitch when they were an early public company. Happy employees == great things. I remember when 60 minutes or dateline did some special about google before 2010 and people were floored by how good the employees got it. However, hiring standards relaxed and now google is slowly rotting from the inside to realize it’s the new IBM.
Long tenure alone is absolutely not an advantage. It’s very easy to have a bunch of dead weight that just looks like they know what’s going on.
So back to your point about experience mattering. That should show up in customer satisfaction, project turnaround speed, success rates, etc. Otherwise it’s pretty meaningless. And guess who already measures those KPIs…
No they don’t. The sales org might but when I was at GOOG the quarterly deadlines had zero influence on launch dates, engineering decisions, etc.
Public companies frequently get punished severely in the market when they have a great quarter but advise a negative outlook for the year.
The perfect example of why that is not the case at least in Silicon Valley is Google and their messengers and social networks... just how many of them exist(ed)? Four? Five? Each of them was someone's promotion project, left to languish, suffer and eventually die afterwards. Honestly it's a wonder Gmail and Maps are still around, their UI hasn't been updated much in ages despite obvious potential for improvements... I guess the only reason why they are alive is that they generate absurd amounts of data.
Granted, but they usually have the option to sell at many points along the way.
Think about an alternative financial instrument. What if you could invest $X in a public company (and get a higher rate of return because of the higher risk) but you lose the ability to sell until five years have passed?
This is one kind of option that might be worth trying. I want to see more mechanisms promoting long-term investment in our public markets. (Bonds are not identical to what I described BTW.)
The lack of these kinds of options (sure, bonds exist but I've not noticed them used with any notable significance in public companies) is why people talk about the short term quarterly focus of Wall Street.
> is why people talk about the short term quarterly focus of Wall Street.
This is a false meme though. Tesla was worth more than multiple profitable long standing car manufacturers combined before it ever turned a profit. Investors fixate on short term performance when the company has no long term vision and deriving cash flows is much easier (e.g. a container ship holding company).
The difference is having the option. :) Behavior tends to be different if you don't have the option to sell during a window of time.
In this hypothetical case it might serve as a built-in grace period. Companies that otherwise might have failed if measured and punished quarter by quarter might have time to build a product over five year time horizon.
It's also not necessarily a disadvantage. The real argument is that people management is important. Everyone from the new person who just joined to the decade long senior engineer need to be held to high standards. Which brings me to my next point.
The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything. I think people routinely underestimate how important good leaders are. "There are no bad teams, only bad leaders" comes to mind. So far Elon has shown himself to be terrible in this regard. With SpaceX and Tesla he communicated a vision, with Twitter he's communicated very little and just instituted chaos.
The problem was even worse: there was no vision at all from no one, not leadership, not the investors, not the users, what Twitter should be, other than "it is a way for instant communication with feeds". Everything else was completely lacking: what features do people want, what moderation policy should be applied, how does Twitter plan to make money.
The only ones that had at least some sort of vision where activists - the left wing, the advertisers and large parts of the users didn't want Nazis any more, and the right wing wanted "free speech" aka allowing Nazis.
Completely agree.
This is true, and in my opinion, true for a reason. And that reason is not "most huge companies are dumb", as opposed to what Musk's cult seem to believe. The reality is, measuring what exactly is "bloat" and precisely cutting that bloat is extremely difficult and firing more than half of your workforce is probably like using a warhammer to do brain surgery.
But while fighting the entropy can be hard, not doing so is almost certain to be lethal.
If an organism is infected by gangrene or cancer (the more extreme forms of entropy for a body), it may be more realistic to cut away whole body parts than to treat it in place, even if there is risk of sudden death.
It seems to em that this is what Musk is trying. Either Twitter will be gone within a year, or they may very well be sustainably profitable within 2-3 years.
If your company is losing money all this time you are likely to be fired eventually in the real world. Job security in sw world had become so high that no one seemed to expect it. Everyone assumed “sure we’re losing money and the company has no direction” but all is fine.
They all stayed there in their tables working for 10 years in a rudderless company as if it was a government job.
The only reason Twitter is in deep financial shit right now is because Elon acquired it in a leveraged buyout and the cost of servicing the debt is estimated around 1B/year.
Something I learned early in my career is that some companies consider this a feature not a bug. As in, they are hoarding talent so that they don't go to work for their competitor.
Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge.
Does Twitter retain enough people to manage Mesos?
Apple is also a big Mesos user, but also moving to k8s.
Then Mesosphere (a company) wanted to bring it to the enterprise market but at the time was competing with Kubernetes... and we all know who won.
> Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica.
> The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers.
https://en.wikipedia.org/wiki/Apache_Mesos
Also, the article seems to suggest Twitter only has two datacenters. That seems surprising for the global reach of the company. Perhaps there are other smaller datacenters that are not prepared to handle the entirety of the site’s traffic.
My current thinking is there’s time to figure out how to operate the current system before it runs into issues that would render it degraded for a prolonged period of time. I noticed TLS certs have already rotated for instance. That was my best guess for simple thing that could fail if managed poorly.
https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc...)
If this is the new town square, you can forward my mail to a cabin in the woods.
What's wrong with the new owner of Twitter using Twitter in exactly the same way that a lot of other Twitter users use it for, memes and jokes?
I find that rather refreshing, actually. It's a variant of eating one's own dogfood.
In my opinion what would be inappropriate for a billionaire CEO, and reveal deep flaws in decision making process, and the decision making process of any government official or governing body who subsequently partners with that CEO, is to fraternize with the likes of Epstein after he was convicted, e.g., https://www.nytimes.com/2019/10/12/business/jeffrey-epstein-.... But I guess morality is also subjective so people might disagree with me there too.
I didn't know that he lead an extremist cult though, not that I have ever cared to find out much about his life or beliefs. It wouldn't surprise me, I think a lot of the ruling class and ultra rich are terrible people who we should not be living under. That said I have noticed the claims of Musk's terribleness and extremism may have a tiny correlation with his unwillingness to toe the Democrat party line. Now quite possibly that's just a coincidence and there's absolutely no causation there, nevertheless if I were ever in a situation where that information would be important to a decision, it would behoove me to take it on its merits rather than on faith.
Maybe the guy is really losing it
I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash.
From what I can tell, in the past week or so,
Are these failures symptomatic of a larger problem, or are they well-isolated parts misbehaving? Can Twitter even experience a failure cascade like Arecibo? Can that be paused/stopped?I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.
Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.
There has been a serious degradation in the quality since the acquisition:
- Sporadically loading tweets - I could go on some tweets and refresh the page multiple times, with tweets fading in and out of existence showing "This tweet is unavailable"
- Tweets that quote tweets of accounts you have blocked behave weird in multiple ways. Sometimes it's just showing a "This tweet is unavailable" instead of "This tweet is from an account you have blocked", and a few interacting with them crashed my timeline on mobile, having to restart the app
- On a few occurences, every third tweet on the timeline was an ad
- 1-2 notifications from crypto spam bots reaching me every day. The same thing previously was filtered out quite reliably (I assume), since then it happened ~once ever 6 months
--------
And those are only the things I've seen personally. Yeah, they are no deal-breakers, and mostly sporadic failures, but it very much feels like a service that is degrading by the day.