At Twitter's scale with the number of API users they had, I can't imagine how they'd filter the signal out of the noise.
This is very much a "someone would have had to know we're using our own API to avoid this disaster" scenario, which is an institutional knowledge problem. The kind of problem that gets hard to solve when XX% of your workforce suddenly left to seek new opportunities.
Ah, good point. Twitter should have been the largest caller.
That 467 error code suggests to me something else is going on and Twitter is emitting the wrong error message (467 isn't a standard HTTP, so I'm wondering if something lower-level emitted a 467 error, it was misinterpreted as an HTTP status, and the error string chooser logic is giving that API access error message due to fall-through).
One of the criticisms against the old twitter engineering team was that they failed to set up proper dev and testing environments. A leaker exposed all of this and it was later confirmed by Musk in a Twitter Spaces call.
> If these systems ever go down, its link rot at scale
Only if one's mental model is that t.co links were ever permanent.
They're privately-owned obfuscator links. Unless someone in the wild was building a mirror mapping, they were always doomed and never to be trusted; they knocked the "U" right out of the "URL."
It's even more-so true for t.co URLs though because instead of just 1 point of failure there are 2 points of failure. The original URL may work for another decade but if you don't know what the URL is behind the t.co URL and Twitter is down or whatever, then you're just out of luck.
I don’t think it necessarily changes the reliability of the URL. A server or dns change is similarly problematic.
The only issues I have with shorteners is that they obfuscate what you’re accessing (so if it changes, it’s harder to know) and they’re a privacy issue
That's what I'm saying. We must reframe our mental models so that everything on Twitter is thought of as temporary. None of it is, or really was ever, stable.
In case people are wondering what this is about: All links and images are not working on twitter right now. Seems like they accidentally included their own services in the api blocklist? Sending best of luck to the engineers. The whole twitter saga has given me a good chunk of valuable "lesson learned" over the last weeks and months.. so thanks for that.
Are you logged in? Twitter seems to be working normally (i.e. it refreshes) where I'm logged into the browser. But where I'm not, I get the error message everybody's seeing.
Definitely region specific I think. It's working mostly fine for me. Links and images are mostly fine except for the errant occasional broken one but even then it's only when opening the full image
Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.
The only cases I can think of for staying:
- Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)
- A feeling one needs the job and can't easily get work elsewhere. Perhaps living paycheck to paycheck, haven't interviewed in awhile, imposter syndrome... (If you're worried you'd not be able to get hired, instead ask why you weren't fired.)
Or we could reduce the hoops & cycle time for legal immigration. The US has an opportunity right now to cement itself as the leader in technology for the next 2-3 decades and stave off the impending population growth crisis if we could find a way to enable people who want to live here to live here.
I'm not interested in government policy that allows companies to bring in cheap foreign labor to undercut my compensation. There are 330 million people in the us, so if companies need more engineers, they can pay more and train more.
Also, there is no population growth crisis. A decreasing population means less stress on the environment and lower housing costs. Yes, there will be economic effects, but it's hardly catastrophic. Japan is managing just fine.
> government policy that allows companies to bring in cheap foreign labor to undercut my compensation
They become Americans and rise to your level of comfort and compensation. By working here in the US, they increase the wealth, tax base, and culture of our country. Immigration is a good thing. Skilled immigration even more so.
The alternative is that they stay in their home country, and that country grows a tech industry that rivals the US. Those workers will work for even cheaper than in the US, putting an even wider delta on price and creating an incredible arbitrage opportunity for talent.
Much like the automotive industry, foreign competition will drive margin out of our comfortable tech industry that has enjoyed being peerless for decades.
We're only going to see more `TikTok`s and `Spotify`s arise.
At some point, talent won't want to come to the US anymore. That should scare you.
I say all of this as someone who wants everyone to enjoy wealth and prosperity regardless of where they live. I still want opportunity and the ability to capitalize on it to be accessible to any American that would take it. And for that to continue, we should keep growing our talent pool and increasing the scope of what we can achieve together.
I see foreign competition as an inevitability, but I don't think number of engineers is a decisive factor. China and India have enough engineers already, and have for a while, but other than Tiktok, no real competition yet. Whereas Sweden, home of Spotify as you pointed out, has very few.
Even if number of engineers is decisive, we will never stop it by immigration. China + India have more than 8x the population of the US. We would never be able to deprive them of enough engineers via immigration.
This is a very xenophobic view on immigration, and arguably the status quo. Past performance does not equate to future results, but I think you can draw the conclusion that the immense diversity that exists in the US compared to "rest of world" is a competitive advantage. What would it mean to lean in?
Yes, there will be economic effects, but it's hardly
catastrophic. Japan is managing just fine.
I'm absolutely unqualified to say if they're correct, but there are a lot of economists predicting disaster for countries like Russia, Japan, and (not far behind them) China who will face this kind of demographic shift. The US is in the midst of this kind of shift as Boomers age through the system, but it is said that it will be less severe (thanks to younger immigrants) and less permanent than some other countries are facing.
Of course, you can find a lot of "experts" saying anything. Economics is an area where I haven't got the chops or the hubris to tell shit from shinola.
Or we could force these tech companies flush with cash to pay market rate and watch salaries rise or let these companies leave the country if they can’t afford to do business here. These are the richest companies on the planet and they don’t need a labor subsidy, they just want one and are powerful enough to get their way.
I was in the industry before this greedy trend accelerated out of control - I know first hand how much better things were before. I’ve seen a colleague choose death over the abuse the average H1B gets subjected to. The sooner this lie ends, the better for all of us.
Maybe folks early in their career. It's not like the cachet has totally dissolved because they did employ a lot of top notch people. Also, when they're interviewing for their next job, even if your departure was super contentious, Twitter is a great scapegoat that nobody will question.
"It says here you were terminated for breaking into a vending machine in the lobby and distributing snacks to your colleagues while dressed as Robinhood."
"Yeah it was an act of protest because management threatened to take away our lunch breaks."
> Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.
I worked for a different eccentric billionaire early in my career. It was stressful, demanding, and unpredictable as he tried to micromanage everything by himself in short bursts of 5 minutes before moving on to the next thing.
The hook was an idea that it was a famous company and he was a [not quite] famous billionaire and that we were sitting on a once in a lifetime opportunity to build our resumes. He was going to reward us greatly at some future date when everything was working well again, and we’d be able to work anywhere else we wanted after this.
Didn’t really pan out for us as everyone got burned out, the billionaire’s micromanagement and constant product churn diminished the company’s reputation, and he eventually laid most people off in favor of even cheaper foreign labor.
But there'll be one or two people who get cozy with Elon and win the lottery and will write articles on how hard work always pays off (neglecting the other 99.9% of Twitter that gets fired when it all gets shipped to India).
Could also be that they have enough money saved up and don't actively need a job, so they're happy to ride this one out until the end and because they don't need the job they don't really get stressed or anything - if they get fired no big deal.
If they actually make it all the way to the end, maybe being the last man standing there actually does have some perks we don't yet know about?
So they don't realize now they are being abused even though they have first hand experience, but you realize they are being abused because you have what insight exactly?
In other words, what exactly do you think the people working there will "realize years later"?
Could you specifically list the abuses being done to these engineers that they are unaware of?
Usually people choose to stay in abusive situations because they don’t believe they’re being abused, or have convinced themselves they have no better option. That doesn’t change the fact the environment is still abusive.
There’s a whole swaths of science and research behind this. It’s why domestic violence can continue for years unknown to anyone, even from friends, until it’s not.
OP isn’t saying they’re being abused as a fact, but if they are — at a workplace that seems as objectively toxic as Twitter right now — many who are still there won’t likely realize they were until afterwards.
They make for a shitty work environment for everyone - anyone who wants to set up healthy boundaries between work and life would be let go, thus eventually reducing worker rights.
No one is in shackles. They make like $300k in compensation. It's a free country, workers can go wherever they want. Free labor market.
What if, just what if, the workers actually like working hard and making their lives better? That seems to never enter the conversation here.
Honestly, the culture on HN makes workers like me more resentful. It's 100% negative, there is absolutely 0% optimism in these threads. Not even one comment here (or elsewhere on HN).
They all make $300k and can all go wherever they want and do whatever they want? Are you certain that is the case?
> ... like working hard and making their lives better
It's true that work can be very rewarding - "Find a job you enjoy doing, and you will never have to work a day in your life" and all that. Plus who doesn't like making their lives better? I guess it is up to the reader to decide whether a job in Musk-era Twitter is one of those jobs, and whether working for the new "hardcore" Twitter would make your life better. I can only speak for myself but if I worked somewhere and I saw that after a takeover:
- half of my colleagues got fired one day
- some people occasionally get fired for on-the-spot violations like answering a CEO's question
- the company went from nearly-profitable to incredibly unprofitable (let’s even ignore how it got that way)
- the company stopped paying some bills and were kicked out of some offices and services
- the owner-and-CEO had become an extremely ... divisive figure, shall we say
- the owner-and-CEO declares that the existing product is brittle and broken and needs a ground-up "2.0" rewrite (a second system so to speak...)
... then I wouldn't see this as some wonderful opportunity, I'd see it as my workplace being turned into a plaything for a tempestuous billionaire who might not be as competent as he wants us to believe. And if I was a wealthy and financially-secure US permanent resident looking at a buoyant jobs market then I'd probably leave, take a bit of a breather then start looking for a new job at a different company. However if any one of those things didn't hold (maybe I'm an H1B visa holder, maybe I am young and don't have much savings or experience, maybe the job market kinda sucks, maybe I have am the sole earner in my family) I'd be in a much different and more depressing situation - maybe not literally "in shackles" but certainly not as free as you are suggesting.
> Did you see the photo of the employee sleeping on the floor?
I’ve done this. I had lots of fun—everyone, from the CEO down, was crunching—and became financially independent after it.
It’s usually a sign of under-resourcing. But almost every start-up experiencing unexpected growth should be ready for it. It’s unusual at Twitter’s scale, but closed to mismanagement than abuse.
Please don’t do that. You deserve better as a worker. If a company is unable to function without infringing on the free time of their workers, that company should not exist.
> People should be allowed to work how they want to
I disagree. There is a race to the bottom and a tragedy of the commons. A company could easily exploit this by gamifying the work and hiring workers with addictive tendencies. Most national labor laws and labor contracts set a maximum number of daily and weekly working hours for a reason. Workplace safety is only one of those reasons.
Yes, and if they do, there is a risk of company exploiting that person and their co-workers for underpaid labor. One worker’s enjoyment of work does not justify this risk.
Nobody, but you must see how this can be abused to do so. A system that can be abused will be abused given the right incentives. As it stands companies are incentivized to pay as little for labor as they can get away with, and if they are allowed to “allow” workers to work excessive overtime, they will. And it will cost every other worker our free time.
If you enjoy your work, that’s great, just do your work during work hours, don’t use your free time to undervalue other workers in the industry.
I think we might be arguing about different things here. You’re saying that it worked for OP, that is fine, I don’t disagree with that, however I do disagree with this practice, because as a whole it is not good for us workers in this industry. This behavior is ripe for abuse, and companies do abuse it unless there is a collective action against it (through regulation, union contracts, etc.)
Like you said, if you work for a company which expects this from you, and you quit as a result, they can simply hire someone else that takes the abuse, and that is not a good prospects for us workers in this industry. Which is the reason I’m asking OP not to do this. Even though they had fun and were handsomely compensated, this behavior does a net disservice for the rest of us.
If a business operates expecting this from employees, that business is being abusive. If a business got into trouble and employees are sacrificing to help it out, that's just an emergency. And it should be exceedingly rare -- if it's not, that's a poorly run business.
Certainly. I've been in a poorly run business that did this, and I quit.
But I don't mind working late every now and during crunch time, it gets paid forward, usually with time off later.
It's about your relationship with your company.
My point was from the sounds of it GP had a good relationship and had fun and no-one on HN should tell them what to do because they don't know the situation.
As far as I know there is shamefully little research on corporate behavior. Most research I hear about has to do with gender pay gap and only in jurisdictions which mandate it (I wonder why this is).
I think our best estimate of corporate behavior is to look at union contracts, and union demands. Almost universally union contracts prohibit workers from working outside of set number of daily, weekly, and monthly hours. Usually there are some clauses for extraordinary circumstances, but I’ve never worked under a union contract which had less than 11 hours mandatory rest between shifts (even in an emergency) which effectively prohibits all nighters. I’ve seen contracts that go down to 7 hours if absolutely necessary (and every hour infringing on 12 hour rest is payed with 90% penalty) but never 0.
Why do unions demand this? My best guess is that this is a common exploitation tactic by corporations, and thusly a common complaint from union members. Thus we can reasonably assume that many (most?) workers who have worked unreasonably long hours have not seen their just benefits for doing so and felt exploited.
Now it would be nice if corporations kept statistics on this, or if there ware regular surveys by governments or impartial research centers so we could get a more accurate picture, but as far as I know, as it stands, this is our best method, and it is not in favor of this practice.
I had equity in the outfit, I made my first million from it, and again, I had fun. If the CEI were at home partying, it would have been a different story. But we were all there. Kidding, coding and trying to keep something we were building together alive.
> you're essentially saying (or it is being interpreted this way by many) "Do the time, it worked for me!"
I’m not saying that. I’m saying doing the time isn’t equivalent to abuse. The facts and circumstances matter, e.g. who is doing it, how often it’s required, and what everyones’ stakes are.
[Raises hand] I spent countless weekends and evenings in my mid-20s to help the company meet a delivery deadlines (earning the company X million pounds each time) or to untangle some production fuckup (saving the company from having to “best price” trades or incur fines). Claims of overtime shushed and dismissed (“come on you already make more than $colleague…”). Promises of equity that were “delivered” in the form of some kind of ultimately worthless “units” that were granted in place of actual shares (just while the legal side is worked out, you know) and which were eventually swept under the carpet as if they never existed.
Those overnighters and the value they created mean very different things to someone who has equity vs the rest of us. As you can tell I’m a little bitter still, I realise now this amounts to wage theft and violates various protections we have in law - I just wish I had known at the time :(
> I’ve done this. I had lots of fun—everyone, from the CEO down, was crunching—and became financially independent after it.
OK, but this doesn't seem relevant to the situations Twitter's (former, in many cases) employees are in at all. There's no quick path to financial independence working at Musk's Twitter--the company is losing money faster than it was when it was public, Musk is cutting costs everywhere, and there are well-documented instances of employees being shown the door after sleeping on the floor.
In general, I'm pretty sure that in the vast majority of cases, people who sleep on the floor at the office do not end up financially independent for it.
> In general, I'm pretty sure that in the vast majority of cases, people who sleep on the floor at the office do not end up financially independent for it.
Doesn’t matter. In the context of software design it’s a well known trope and most who do it enjoy the work:
There's plenty of evidence on Twitter, from Musk firing engineers who confront him with straight technical questions, to Musk demanding ridiculous work hours as if it's a sign of strength or loyalty.
I can say from experience it's possible to be in an abusive work environment and not decide to quit (through some combination of unawareness of how bad things are, fear of having given up, not wanting to re-enter the job market, etc)
It doesn't make sense from the outside (or in hindsight), but it happens
I'm struggling to see their POV on that. The deal got them $54 for their options, which is a much better deal (probably by more than 2x) than they would have got anywhere else.
For someone who has tried to avoid hearing as much as possible about this drama (and didn't notice this was about Twitter until looking at the comments), what are the condensed "lesson learned" takeaways if you don't mind elaborating?
He created that drama when he tried to gaslight the entire tech industry into thinking 90% of engineering jobs were redundant, you can’t expect anything else than for him to get dunked on when the completely predictable consequences of that play out, namely that the service will get more and more unreliable over time.
If it’s nothing, why is Musk on Twitter saying the whole codebase is brittle for no reason, and needs a complete rewrite? Sounds like a pretty big deal to me.
As far as the drama comment, you’re free to disengage from this thread at any time. No need to continually interject how against the drama you are. Just leave if you find it distasteful.
A 1 hour down time every infrequently is nothing, but at the same time a codebase can be brittle. Are you saying the two are mutually exclusive?
Brittle code means it's hard to change, and causes issues when changed, but other than that it's stable when running.
And I have every right to point out the drama here. People here do somewhat have the right to create drama, but it is distasteful and unprofessional and I will continue to call it out.
Glad for you to admit there is drama here though :)
The frequency and types of outages and failures is significantly more frequent now than pre-Elon. This isn't a surprise to anyone, given Elon's strategy for maintaining Twitter (or not maintaining it, as the case may be).
I don't interpret a lot of "drama" as you put it, but interest. Many observers here are in this field, and follow the "chaos engineering" discipline. Some of them use tools like "chaos monkey" that simulates a metaphorical monkey running through your server room turning off random things, to see how well your resilience systems cope. It's a rare and greatly interesting sight to such practitioners to get to see what happens when the monkey "disconnecting the more sensitive server racks" is a more literal one.
>What was the reason when Twitter went down pre-Elon?
To use a tennis metaphor, good players minimize unforced errors and recover quickly from forced errors.
This is a very clear unforced error that could likely have been prevented by just waiting to roll out the new feature.
To extend the tennis metaphor, it would be like Serena Williams losing a set on 50 double faults. Sure, she's lost other sets before, but it would be notable for her to lose in such a unique way, even if she still went on to win the match.
That's why people are talking about this, it's a very weird way for a site to fail and it's interesting how it happened.
I honestly think the engineering/ops problems are the least of their failures. They bled revenue not because of outages, but because wild policy swings and chaotic management style alienated some of their biggest customers.
If they had frozen features and left the existing policies in tact, I suspect we would have a dramatically different narrative about the layoffs. If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Instead, though, we have a company in crisis due to its mismanagement of other areas, so we're primed to view stuff like this through the lens of that broader failure.
> If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Overstaffed in order to maintain Twitter as a static service that never ships new things, sure.
I guess they've been able to ship some things that the old Twitter had already implemented and/or a/b tested. But I'm not sure those count. Meanwhile people have been paying in advance for Twitter Blue features that were promised 3 months ago.
- When you fire a lot of engineers, legal, security experts... things will still somehow keep working
- Also some people still want to keep working at this place (or must, because of VISA etc); It probably happens all the time, just this time everyone was watching it in full motion.
- I was sure this whole takeover was the end of twitter, but somehow with how many users twitter has, it just won't die. Goes to show the advantages of the mass.. probably the same applies to microsoft/google..
- The whole 8$ for a checkmark story.. which destroyed the whole trust in the checkmark basically immediately. The whole verification process (how that worked) - many people shared some insights on that and how much effort it was to fine-tune all of that with the scale of twitter.
- The free API just being shut down with basically zero communication to devs. and somehow getting away with it. I guess this is a reoccuring theme in tech that a small company has to be open for developers until it's big enough to turn on them.
- With the reduced engineering and the new management, it seems more errors slip into production (and incidents like the one now) - or maybe it just feels like it with all the focus on twitter right now - but anyways, you just see things breaking you normally would not expect to see..
- How people are communicating, or in this case not communicating.
I'm probably not the ideal person to write this down, there was so much stuff going on, some people probably made a whole blog-article series on this. This is just a few of the things that I'm able to remember right now.. and with everything on hn here it gives you some ideas and things to think about. Hope that Helps.
Maybe mister "genius" is working his magic by annoying as many people as he can and making Twitter visibly worse before it is finally great again! So people will finally be amazed at how "genius" he really is.
Seems to be affecting web view but not mobile clients. Wonder if this has anything to do with them not paying their AWS bill as reported the other day.
This happened right as I was using Selenium to log in to Twitter for a project. Was wondering what the hell I was doing wrong until I saw this and realized it wasn't just me.
I don't spend enough time on Reddit for that to really resonate, but the badges and icons flying all over the place and talk of Twitter Gold (or whatever they're calling it) do feel very Reddit-esque these days.
> And when it's not crashed it's slow as cold shit.
And stale. When they changed algos in ~2016(?) to try and stop certain subs I swear it got stale and feels like it updates on a cron job once a day anymore.
I’m in what is likely the top 1% of Reddit users by volume (I have a problem, send help), and I have absolutely no clue what you’re talking about.
Further, as a dev of 15 years, I’m not even sure what you mean by “crash” here. Are you saying that Reddit serves exclusively 500s or timeouts for a sustained period of time on a weekly basis? Or that it serves more than zero 500s at some point during a given week? Or that you, personally, encounter an issue of any kind over a week?
So many ways to describe “crashes” that yeah, by some definitions it probably does, but “can’t use the service” such as what Twitter is experiencing, I don’t think that’s accurate.
One of the triggers used to be that if a thread got too many comments it would bring down the website. So sports communities had to start splitting up their game threads during massive events so Reddit wouldn't break.
It definitely used to happen more often, but it was doing it for me yesterday coincidentally.
Things fail slower then faster than people expect.
Lots of people were waiting with baited breath for Twitter to fail the first week Musk bought it, but most of the failure conditions had been automated away...
Instead the reliability loss creep happened more slowly. The people that understood the edge conditions in the system were fired. Then new changes need implemented and the old, now not understood systems, were ripped out and new pieces put in. And lo and behold they have tons of failure conditions and edge cases that can knock out the system.
Even if your programming team wrote immaculate documentation on all these edge conditions, it can take an immense amount of time to both read them, and then fully understand them, and when you have a micromanager breathing down your neck for changes 10 minutes ago this is what happens.
Local status updates and the occasional tidbit of interest from people whose thoughts I like to read but who twitterblog instead of using an actual blog.
All still present (positive and negative) on Twitter, plus the benefit to open-minded people of hearing opinions from all sides, which has improved post-Elon.
I used to run a bot that broadcasted my town's current alternate side parking rules (for snow removal operations) but after twitter got weird the town decided to make a text messaging service for it.
I still use it to follow certain artists but if that subcommunity moves I'll follow them elsewhere.
That second one, poking fun at the BBC for saying that Twitter "can't protect users from trolling," was really the BBC shooting itself in the foot with a laughable headline burying the lede of the article:
> Twitter insiders: We can't protect users from trolling under Musk
> Twitter insiders have told the BBC that the company is no longer able to protect users from trolling, state-co-ordinated disinformation and child sexual exploitation, following lay-offs and changes under owner Elon Musk. [1]
For some reason, the headline decided to focus on "trolling" over an increase in child sexual assault and state-co-ordinated disinformation...
Honestly the "insider" even talking about "protecting" people from trolling and state co-ordinated misinformation(?) in the same sentence as protecting them from csam makes it very hard to believe it's something genuine.
Sounds like the people who say all the petty things they dislike about someone on the internet and then finishes with "he's a pedo" to add gravitas.(yes elon does fall into this category)
It not just the layoffs it is also the major changes Elon is forcing through at high speed .
Hiring for twitter is going to be quite expensive, nobody is taking this up without huge payoff and probably upfront given Elon pre disposition not to honor severance etc
> half the time stuff isn't loading at all or is broken in obvious, overt ways
I use the app for a couple hours a day (according to my iPhone weekly reports), and have barely noticed any degradation since "the takeover". (Notwithstanding the obvious outage happening currently).
There's been a few minor glitches for sure, but then what large service is immune? Facebook, Exchange, AWS, etc. have all had notable outages this past year, and the Apple Services I regularly use (Apple Music, Siri, etc.) "fail" more often for me in daily use.
Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
> Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
In my ideal world, Twitter - or something similar - succeeds. I don't care much about the politics of the ownership because that's generally going to be an iffy subject with anything coming from the valley.
But to regularly use it and not see things breaking here and there every day (not even including the times when the API is borked or it's only recommending Elon Tweets in your feed or a person's timeline won't load no matter what, not being able to send a tweet, etc.) ... I have to imagine you're very lucky. There was about 11 hours when people had exceeded their daily tweet quota by trying to send a single tweet. You don't remember stuff like that? It happens multiple times a week in the last few months.
I'm certainly not doing anything advanced. I'm mostly a casual user.
I can corroborate. Videos struggle to load, timeline doesn't refresh unless you pull down multiple times, my current position gets lost multiple times forcing me to scroll down and then scroll back up, tweets showing up multiple times, many more ads than before and so on
It's really been kind of miserable to use the app and I only open it once or twice a day. It had issues before but never all at once like this
I've never seen a large company whose entire existence is a single app perform so poorly. It was down for me a whole day a week or so ago. Bugs are pretty rampant, they're small, but didn't happen to me previously. It's just been gradual degradation for months now. Idk how you aren't noticing.
As someone that I think the previous twit sphere would accuse of being right wing adjacent, I’m shocked at the dumpster fire that twitter has become from a content standpoint.
Like.. wokeness was annoying but the trash heap of craziness it is today is 5 steps backward. My feed is nothing but conspiratorial nonsense.
If twitter was “tilted left” before, today it seems to have capsized rightward.
Right, it turns out that bringing back a bunch of previously banned trolls will chase more centrist people away, leaving only the trolls and those who like to argue with them. Not sure if there have been algorithm tweaks as well (aside from the obnoxious removal of the sort by latest button Elon swore he'd keep), my feed is the same. Combined with the stupid "Trending" tweaks and it's a worse version of Reddit.
I honestly can't believe that not only did he welcome back actual Nazis that were previously banned, but he sold them checkmarks that let them have outsized influence.
Elon's own account is that he's "experimenting" with taking production services offline to cut cost and simplify their code base. Essentially a literal Chaos Monkey. With this in mind, service degradations aren't surprising.
I still don't understand why every fifth post in my feed is an Elon tweet, I don't even follow him. I could block him but I find it interesting to see how much of his content is pushed toward me who hasn't asked for it.
Maybe better to block him. I did so because I was a bit sick of seeing his tweets - some of them are borderline offensive, some are him cosying up with far-right but the majority are just a bit try-hard, unfunny and ultimately sad.
I'm not a delicate flower when it comes to offense so I'd just roll my eyes when the first two kinds of tweets popped up. But seeing a grown man repeatedly trying and failing so hard at being a poster is a bit much to take.
491 comments
[ 3.0 ms ] story [ 299 ms ] threadAt Twitter's scale with the number of API users they had, I can't imagine how they'd filter the signal out of the noise.
This is very much a "someone would have had to know we're using our own API to avoid this disaster" scenario, which is an institutional knowledge problem. The kind of problem that gets hard to solve when XX% of your workforce suddenly left to seek new opportunities.
That 467 error code suggests to me something else is going on and Twitter is emitting the wrong error message (467 isn't a standard HTTP, so I'm wondering if something lower-level emitted a 467 error, it was misinterpreted as an HTTP status, and the error string chooser logic is giving that API access error message due to fall-through).
Given all the random things Elon has cut from Twitter, it wouldn't surprise me if that included a staging environment.
https://twitter.com/emollick/status/1632788892599586816
> The Twitter outage seems to be around a critical piece of infrastructure: the Twitter link shortener.
> Every link on Twitter has been shortened to a t.co link. Only Twitter’s systems know where the link goes
> If these systems ever go down, its link rot at scale
Only if one's mental model is that t.co links were ever permanent.
They're privately-owned obfuscator links. Unless someone in the wild was building a mirror mapping, they were always doomed and never to be trusted; they knocked the "U" right out of the "URL."
The only issues I have with shorteners is that they obfuscate what you’re accessing (so if it changes, it’s harder to know) and they’re a privacy issue
Seems to be related to a query param in my case (if I remove the ?lang=en, the page loads properly).
... mostly around the hope they can jump ship before the Titanic pulls them under.
The only cases I can think of for staying:
- Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)
- A feeling one needs the job and can't easily get work elsewhere. Perhaps living paycheck to paycheck, haven't interviewed in awhile, imposter syndrome... (If you're worried you'd not be able to get hired, instead ask why you weren't fired.)
Or we could reduce the hoops & cycle time for legal immigration. The US has an opportunity right now to cement itself as the leader in technology for the next 2-3 decades and stave off the impending population growth crisis if we could find a way to enable people who want to live here to live here.
Also, there is no population growth crisis. A decreasing population means less stress on the environment and lower housing costs. Yes, there will be economic effects, but it's hardly catastrophic. Japan is managing just fine.
They become Americans and rise to your level of comfort and compensation. By working here in the US, they increase the wealth, tax base, and culture of our country. Immigration is a good thing. Skilled immigration even more so.
The alternative is that they stay in their home country, and that country grows a tech industry that rivals the US. Those workers will work for even cheaper than in the US, putting an even wider delta on price and creating an incredible arbitrage opportunity for talent.
Much like the automotive industry, foreign competition will drive margin out of our comfortable tech industry that has enjoyed being peerless for decades.
We're only going to see more `TikTok`s and `Spotify`s arise.
At some point, talent won't want to come to the US anymore. That should scare you.
I say all of this as someone who wants everyone to enjoy wealth and prosperity regardless of where they live. I still want opportunity and the ability to capitalize on it to be accessible to any American that would take it. And for that to continue, we should keep growing our talent pool and increasing the scope of what we can achieve together.
Even if number of engineers is decisive, we will never stop it by immigration. China + India have more than 8x the population of the US. We would never be able to deprive them of enough engineers via immigration.
Of course, you can find a lot of "experts" saying anything. Economics is an area where I haven't got the chops or the hubris to tell shit from shinola.
Be careful what you wish for.
Clearly not for everyone given the amount of opting out and controversy.
"It says here you were terminated for breaking into a vending machine in the lobby and distributing snacks to your colleagues while dressed as Robinhood."
"Yeah it was an act of protest because management threatened to take away our lunch breaks."
"Well... it... sounds plausible I guess."
I worked for a different eccentric billionaire early in my career. It was stressful, demanding, and unpredictable as he tried to micromanage everything by himself in short bursts of 5 minutes before moving on to the next thing.
The hook was an idea that it was a famous company and he was a [not quite] famous billionaire and that we were sitting on a once in a lifetime opportunity to build our resumes. He was going to reward us greatly at some future date when everything was working well again, and we’d be able to work anywhere else we wanted after this.
Didn’t really pan out for us as everyone got burned out, the billionaire’s micromanagement and constant product churn diminished the company’s reputation, and he eventually laid most people off in favor of even cheaper foreign labor.
But there'll be one or two people who get cozy with Elon and win the lottery and will write articles on how hard work always pays off (neglecting the other 99.9% of Twitter that gets fired when it all gets shipped to India).
They can switch jobs. H-1B doesn't force you to stay at your job.
You forgot the bootlickers, I'm sure not all of them have been fired yet.
If they actually make it all the way to the end, maybe being the last man standing there actually does have some perks we don't yet know about?
In other words, what exactly do you think the people working there will "realize years later"?
Could you specifically list the abuses being done to these engineers that they are unaware of?
There’s a whole swaths of science and research behind this. It’s why domestic violence can continue for years unknown to anyone, even from friends, until it’s not.
OP isn’t saying they’re being abused as a fact, but if they are — at a workplace that seems as objectively toxic as Twitter right now — many who are still there won’t likely realize they were until afterwards.
I simply asked OP to list the abuses they think the engineers are unaware of.
> OP isn’t saying they’re being abused as a fact,
OP literally said "It would feel amazing after this much abuse"
> they will _probably realize_ years later that they were abused.
Their "probably" was about them probably realizing, not probably being abused.
OP was absolute in their accusation of abuse in his above post that I quoted.
That's why I asked them for specific examples.
This much abuse. What abuse? Can you tell us more in detail?
What if, just what if, the workers actually like working hard and making their lives better? That seems to never enter the conversation here.
Honestly, the culture on HN makes workers like me more resentful. It's 100% negative, there is absolutely 0% optimism in these threads. Not even one comment here (or elsewhere on HN).
> ... like working hard and making their lives better
It's true that work can be very rewarding - "Find a job you enjoy doing, and you will never have to work a day in your life" and all that. Plus who doesn't like making their lives better? I guess it is up to the reader to decide whether a job in Musk-era Twitter is one of those jobs, and whether working for the new "hardcore" Twitter would make your life better. I can only speak for myself but if I worked somewhere and I saw that after a takeover:
- half of my colleagues got fired one day
- some people occasionally get fired for on-the-spot violations like answering a CEO's question
- the company went from nearly-profitable to incredibly unprofitable (let’s even ignore how it got that way)
- the company stopped paying some bills and were kicked out of some offices and services
- the owner-and-CEO had become an extremely ... divisive figure, shall we say
- the owner-and-CEO declares that the existing product is brittle and broken and needs a ground-up "2.0" rewrite (a second system so to speak...)
... then I wouldn't see this as some wonderful opportunity, I'd see it as my workplace being turned into a plaything for a tempestuous billionaire who might not be as competent as he wants us to believe. And if I was a wealthy and financially-secure US permanent resident looking at a buoyant jobs market then I'd probably leave, take a bit of a breather then start looking for a new job at a different company. However if any one of those things didn't hold (maybe I'm an H1B visa holder, maybe I am young and don't have much savings or experience, maybe the job market kinda sucks, maybe I have am the sole earner in my family) I'd be in a much different and more depressing situation - maybe not literally "in shackles" but certainly not as free as you are suggesting.
I’ve done this. I had lots of fun—everyone, from the CEO down, was crunching—and became financially independent after it.
It’s usually a sign of under-resourcing. But almost every start-up experiencing unexpected growth should be ready for it. It’s unusual at Twitter’s scale, but closed to mismanagement than abuse.
I've pulled late nights, they bought us pizza and gave us a half day later. Sometimes you have to put in work during critical times.
It's certainly not construction work anyway, people are kinda soft here sometimes.
I disagree. There is a race to the bottom and a tragedy of the commons. A company could easily exploit this by gamifying the work and hiring workers with addictive tendencies. Most national labor laws and labor contracts set a maximum number of daily and weekly working hours for a reason. Workplace safety is only one of those reasons.
If you enjoy your work, that’s great, just do your work during work hours, don’t use your free time to undervalue other workers in the industry.
Yes, shit companies exist and abuse their employees. You quit and move on. There's no indication that this was happening.
Like you said, if you work for a company which expects this from you, and you quit as a result, they can simply hire someone else that takes the abuse, and that is not a good prospects for us workers in this industry. Which is the reason I’m asking OP not to do this. Even though they had fun and were handsomely compensated, this behavior does a net disservice for the rest of us.
If a business operates expecting this from employees, that business is being abusive. If a business got into trouble and employees are sacrificing to help it out, that's just an emergency. And it should be exceedingly rare -- if it's not, that's a poorly run business.
But I don't mind working late every now and during crunch time, it gets paid forward, usually with time off later.
It's about your relationship with your company.
My point was from the sounds of it GP had a good relationship and had fun and no-one on HN should tell them what to do because they don't know the situation.
Shall we source our percentages or just keep throwing them out?
I think our best estimate of corporate behavior is to look at union contracts, and union demands. Almost universally union contracts prohibit workers from working outside of set number of daily, weekly, and monthly hours. Usually there are some clauses for extraordinary circumstances, but I’ve never worked under a union contract which had less than 11 hours mandatory rest between shifts (even in an emergency) which effectively prohibits all nighters. I’ve seen contracts that go down to 7 hours if absolutely necessary (and every hour infringing on 12 hour rest is payed with 90% penalty) but never 0.
Why do unions demand this? My best guess is that this is a common exploitation tactic by corporations, and thusly a common complaint from union members. Thus we can reasonably assume that many (most?) workers who have worked unreasonably long hours have not seen their just benefits for doing so and felt exploited.
Now it would be nice if corporations kept statistics on this, or if there ware regular surveys by governments or impartial research centers so we could get a more accurate picture, but as far as I know, as it stands, this is our best method, and it is not in favor of this practice.
I worked some 16 hour shifts (got OT) and I regret it.
I had equity in the outfit, I made my first million from it, and again, I had fun. If the CEI were at home partying, it would have been a different story. But we were all there. Kidding, coding and trying to keep something we were building together alive.
And they should have recourse. That doesn’t mean we ban or automatically condone the practice.
I’m not saying that. I’m saying doing the time isn’t equivalent to abuse. The facts and circumstances matter, e.g. who is doing it, how often it’s required, and what everyones’ stakes are.
Those overnighters and the value they created mean very different things to someone who has equity vs the rest of us. As you can tell I’m a little bitter still, I realise now this amounts to wage theft and violates various protections we have in law - I just wish I had known at the time :(
"Now is the time you want to work hard ... trust me"
"It isn't in the budget now, but we're just about to close ... "
"If we pull through this, everyone here will be rewarded!"
"Now isn't the time to leave ... trust me"
It's shameless!
OK, but this doesn't seem relevant to the situations Twitter's (former, in many cases) employees are in at all. There's no quick path to financial independence working at Musk's Twitter--the company is losing money faster than it was when it was public, Musk is cutting costs everywhere, and there are well-documented instances of employees being shown the door after sleeping on the floor.
In general, I'm pretty sure that in the vast majority of cases, people who sleep on the floor at the office do not end up financially independent for it.
Doesn’t matter. In the context of software design it’s a well known trope and most who do it enjoy the work:
https://thehill.com/policy/technology/3876505-twitter-employ...
Only the company would really have insight into that.
Pulling a late night and sleeping on the ground does not make you immune to a layoff.
It doesn't make sense from the outside (or in hindsight), but it happens
Unfortunately, there are his blind followers trying to emulate him. Not sure what havoc they are planning to unleash on their employees.
https://www.businessinsider.com/marc-benioff-elon-musk-ceos-...
paywall-less link - https://archive.ph/HbC3k
Before, it was "healthy for everyone" when Twitter went down (search hn.algolia.com for pre-Elon Twitter downtime threads)
Now it's an episode of Real Housewives just like any other Elon thread on HN.
No need to pull out terms like gaslighting to justify creating more drama.
A 1 hour downtime is nothing.
If it’s nothing, why is Musk on Twitter saying the whole codebase is brittle for no reason, and needs a complete rewrite? Sounds like a pretty big deal to me.
As far as the drama comment, you’re free to disengage from this thread at any time. No need to continually interject how against the drama you are. Just leave if you find it distasteful.
Brittle code means it's hard to change, and causes issues when changed, but other than that it's stable when running.
And I have every right to point out the drama here. People here do somewhat have the right to create drama, but it is distasteful and unprofessional and I will continue to call it out.
Glad for you to admit there is drama here though :)
Perhaps the takeaway is that we should get our shit together. The opposite of r/antiwork.
What was the reason when Twitter went down pre-Elon?
I really don't see the reason for this artificial drama. Do other "X service is down" threads go this way?
People say Elon is dramatic, but this thread is honestly ridiculous and way more dramatic than anything I've seen him post.
I don't interpret a lot of "drama" as you put it, but interest. Many observers here are in this field, and follow the "chaos engineering" discipline. Some of them use tools like "chaos monkey" that simulates a metaphorical monkey running through your server room turning off random things, to see how well your resilience systems cope. It's a rare and greatly interesting sight to such practitioners to get to see what happens when the monkey "disconnecting the more sensitive server racks" is a more literal one.
To use a tennis metaphor, good players minimize unforced errors and recover quickly from forced errors.
This is a very clear unforced error that could likely have been prevented by just waiting to roll out the new feature.
To extend the tennis metaphor, it would be like Serena Williams losing a set on 50 double faults. Sure, she's lost other sets before, but it would be notable for her to lose in such a unique way, even if she still went on to win the match.
That's why people are talking about this, it's a very weird way for a site to fail and it's interesting how it happened.
Musk's cost-cutting may yet pan out from a business perspective, but it seems to be a pretty risky move.
If they had frozen features and left the existing policies in tact, I suspect we would have a dramatically different narrative about the layoffs. If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Instead, though, we have a company in crisis due to its mismanagement of other areas, so we're primed to view stuff like this through the lens of that broader failure.
Overstaffed in order to maintain Twitter as a static service that never ships new things, sure.
I guess they've been able to ship some things that the old Twitter had already implemented and/or a/b tested. But I'm not sure those count. Meanwhile people have been paying in advance for Twitter Blue features that were promised 3 months ago.
I'm probably not the ideal person to write this down, there was so much stuff going on, some people probably made a whole blog-article series on this. This is just a few of the things that I'm able to remember right now.. and with everything on hn here it gives you some ideas and things to think about. Hope that Helps.
https://twitter.com/
Just a steady degradation of service.
I don't spend enough time on Reddit for that to really resonate, but the badges and icons flying all over the place and talk of Twitter Gold (or whatever they're calling it) do feel very Reddit-esque these days.
And stale. When they changed algos in ~2016(?) to try and stop certain subs I swear it got stale and feels like it updates on a cron job once a day anymore.
Further, as a dev of 15 years, I’m not even sure what you mean by “crash” here. Are you saying that Reddit serves exclusively 500s or timeouts for a sustained period of time on a weekly basis? Or that it serves more than zero 500s at some point during a given week? Or that you, personally, encounter an issue of any kind over a week?
So many ways to describe “crashes” that yeah, by some definitions it probably does, but “can’t use the service” such as what Twitter is experiencing, I don’t think that’s accurate.
https://www.reddit.com/r/OutOfTheLoop/comments/2trmnk/why_is...
(See: all the people agreeing)
One of the triggers used to be that if a thread got too many comments it would bring down the website. So sports communities had to start splitting up their game threads during massive events so Reddit wouldn't break.
It definitely used to happen more often, but it was doing it for me yesterday coincidentally.
At least once a week i'm met with the 'trippin' out snoo for a few minutes.
Lots of people were waiting with baited breath for Twitter to fail the first week Musk bought it, but most of the failure conditions had been automated away...
Instead the reliability loss creep happened more slowly. The people that understood the edge conditions in the system were fired. Then new changes need implemented and the old, now not understood systems, were ripped out and new pieces put in. And lo and behold they have tons of failure conditions and edge cases that can knock out the system.
Even if your programming team wrote immaculate documentation on all these edge conditions, it can take an immense amount of time to both read them, and then fully understand them, and when you have a micromanager breathing down your neck for changes 10 minutes ago this is what happens.
[0] e.g. Elon: https://twitter.com/elonmusk
Easy access to information and articles about breaking news events.
Community.
A feed of interesting articles and discussions.
Enjoyment gained from following quickly evolving memes, discussions, and jokes.
A way to verbalize incidental thoughts, and the pleasure of crafting those thoughts into small nuggets to share with my friends.
Of course there are many negative aspects to Twitter, but those are the things I found value in.
I still use it to follow certain artists but if that subcommunity moves I'll follow them elsewhere.
There are a million interesting discussions to be had in the world today, but he's stuck on mediocre Reddit shitposting from like 2012.
[1] https://twitter.com/elonmusk/status/1632574090216103937
[1] https://twitter.com/elonmusk/status/1632574090216103937
[2] https://twitter.com/elonmusk/status/1632682409366781953
[3] https://twitter.com/elonmusk/status/1632275726912110593
> Twitter insiders: We can't protect users from trolling under Musk
> Twitter insiders have told the BBC that the company is no longer able to protect users from trolling, state-co-ordinated disinformation and child sexual exploitation, following lay-offs and changes under owner Elon Musk. [1]
For some reason, the headline decided to focus on "trolling" over an increase in child sexual assault and state-co-ordinated disinformation...
1. https://www.bbc.com/news/technology-64804007
Sounds like the people who say all the petty things they dislike about someone on the internet and then finishes with "he's a pedo" to add gravitas.(yes elon does fall into this category)
I guess there's still one aspect of the company that Musk hasn't "fixed" yet...
I barely use the app anymore and half the time stuff isn't loading at all or is broken in obvious, overt ways.
Just ... stunning.
Hiring for twitter is going to be quite expensive, nobody is taking this up without huge payoff and probably upfront given Elon pre disposition not to honor severance etc
Either he shuts it down at the point or he has to spend a lot of money .
IMO More likely he shuts down and blame any range of external factors were conspiring to take him down rather than accept he decisions were wrong
I use the app for a couple hours a day (according to my iPhone weekly reports), and have barely noticed any degradation since "the takeover". (Notwithstanding the obvious outage happening currently).
There's been a few minor glitches for sure, but then what large service is immune? Facebook, Exchange, AWS, etc. have all had notable outages this past year, and the Apple Services I regularly use (Apple Music, Siri, etc.) "fail" more often for me in daily use.
Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
In my ideal world, Twitter - or something similar - succeeds. I don't care much about the politics of the ownership because that's generally going to be an iffy subject with anything coming from the valley.
But to regularly use it and not see things breaking here and there every day (not even including the times when the API is borked or it's only recommending Elon Tweets in your feed or a person's timeline won't load no matter what, not being able to send a tweet, etc.) ... I have to imagine you're very lucky. There was about 11 hours when people had exceeded their daily tweet quota by trying to send a single tweet. You don't remember stuff like that? It happens multiple times a week in the last few months.
I'm certainly not doing anything advanced. I'm mostly a casual user.
It's really been kind of miserable to use the app and I only open it once or twice a day. It had issues before but never all at once like this
Like.. wokeness was annoying but the trash heap of craziness it is today is 5 steps backward. My feed is nothing but conspiratorial nonsense.
If twitter was “tilted left” before, today it seems to have capsized rightward.
I honestly can't believe that not only did he welcome back actual Nazis that were previously banned, but he sold them checkmarks that let them have outsized influence.
I still don't understand why every fifth post in my feed is an Elon tweet, I don't even follow him. I could block him but I find it interesting to see how much of his content is pushed toward me who hasn't asked for it.
I'm not a delicate flower when it comes to offense so I'd just roll my eyes when the first two kinds of tweets popped up. But seeing a grown man repeatedly trying and failing so hard at being a poster is a bit much to take.
https://fortune.com/2023/03/02/elon-musk-bernard-arnault-wor...