This kind of attitude comes up on HN a lot. They have lots of internal tooling and jobs that have to be done, especially at their scale.
WhatsApp with 50 employees is an anomaly. In reality, companies need lots more employees just to get by. Not all industries are equal when it comes to responsibilities and programmers are the most ignorant but arrogant class, thinking that all you need is software. Spolsky covered this 2 decades ago in his “Developer Abstraction Layer” article.
they also haven't made a single dime in what, 11-ish years? Writing software is fun, but usually companies do it to earn money. If they were building the death star it would be one thing, but they're making a taxi app. The entire company feels like a jobs program for software engineers run by Softbank instead of the government
or you turn out to be the next MoviePass because selling a dollar for 75 cents isn't a business. I would not make the mistake and compare everything that grows fast to Amazon or Netflix. The former has logistic centres and half of the servers that run the internet, the latter a ton of content. It's not clear to me what Uber actually owns. They even sold off their self driving unit.
Don’t get so caught up with what a company “owns”, what matters is what they do. Owning something isn’t a guarantee that it won’t become a worthless asset later.
Also MoviePass was at a good start by selling tickets at a loss, what killed them was that they couldn’t leg into the profit generating phase of their strategy and burnt out.
And all the software to support and enable users to make use of it... as well as a ton of cutting edge research in the fields of IC design and manufacturing, computer graphics and artificial intelligence.
But it's not "in reality". It's "with the mindsets and experience and architecture and tooling that are prevalent in the industry"...that's what takes more.
Any given project might not require large numbers people, the headcount driver might the process.
Where you have control, would recommend rethinking the model to look more like Whatsapp than to dismiss as an anomaly.
I would guess a sizable fraction is regional policy, legal and marketing teams. The whole point of Uber is to operate everywhere and you simply can't run all of that in Silicon Valley without access to local knowledge. Not to mention the incredible bloat that comes from building in market-specific features across the whole planet.
It would be handy to understand how many of those 21,000 employees are local operations vs centralised ops (like development).
I'm sure it takes more developers to build and support Uber than you might initially guess, but I'm also sure that Uber just has more employees than they need "because they're successful".
Not surprisingly, software engineers think that the only thing going on with a company like Uber is software engineering. Which is way more complex and requires way more people than it is frequently given credit for. (Uber isn't just an app; it's a bunch of apps, supporting a 2-way marketplace with complex dynamics and many millions of participants.)
But the vast majority of Uber's employees are not software engineers. Running a physical service in ~70 countries is /hard/ and requires a lot of people. Support and operations eat up a lot of the headcount, and can't be scaled like software. Similarly you need a lot of accountants and lawyers to operate globally, along with all of the infrastructure to support a global workforce.
They're doing complicated things, but is that headcount in proportion to it? For comparison, Apple's HQ headcount is under 20k, and entire state level governments for small to mid-sized states are in the low tens of thousands of employees. We all know human things generate lots of work to employ lots of people, but the scale still feels a bit off, even if no one can point to exactly why.
It doesn't feel off to me. As others have discussed in considerable detail, operating in a lot of countries worldwide is a lot more complex than it intuitively seems to be, and the complexity scales superlinearly.
I suspect they might be a few percent overstaffed, but I don't think that hurts a company focusing on growth over all else, and I don't think it's even all that uncommon for such companies. It wouldn't surprise me to see a round or two of layoffs once that growth tops out, but it would surprise me if they added up to more than 10% of headcount.
the two largest employers are both the number one players in their market, being the number 1 player in a market may require that you invest for "first mover advantage". This would lead to a lot of failed/re-tooled initiatives over time.
Being number 2 often just means copying the number 1 player in a more capital efficient manner, product descriptions become "X did it this way, customers want it this way". In a winner takes all market, being the first mover matters a lot.
Also, diminishing returns. In many markets you can "get to the table" with a small crew, but to actually win the customer you'll need bodies almost entirely dedicated to it. And it's that sort of margin, multiplied over and over, that makes one company the undisputed n.1.
Where did you get the 100 number from? I think Nintendo is an interesting company for comparison here but I do find it hard to believe they only had 100 employees as recently as 2011.
I work for the a huge financial institution, its the largest exchange by volume, we operate in multiple markets all across the globe and have our trading platforms distributed across many locations such as NYC, London and Tokyo in datacenters we own, manage and operate, our matching engine matches more daily trades than there are daily Tweets all with a guaranteed sub 50 microseconds latency with 99th percentile sitting around 20 microseconds.
We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.
We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.
We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
> If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
What products are exclusively traded on your exchange? If NASDAQ or NYSE goes down for half a day it’s just an article on WSJ at best. They shut down every night and for entire days on holidays. What financial exchange truly has a meaningful impact when it goes down?
Derivatives and futures a lot of them are exclusive contracts not available on other exchanges, if we go down during trading hours we take the market with us.
Equities aren’t a real economy choke point. They are also, as you observe, federated.
A more critical and delicate financial artery is the tri-party repo marker. If BoNY Mellon or JPMorgan’s repo desks went offline, it would almost certainly shut down large swathes of the American economy, prompt a credit crisis and merit urgent Federal Reserve and possibly Congressional intervention.
But what is the timeline here? If JPM goes down for 2 hours it will cause a congressional intervention? That’s honestly really hard to believe and it borders in delusions of grandeur. Congress intervenes after days at a minimum of an ongoing crisis.
My point is that your strange Internet dick measuring contest of micros to match up an order book is only one tiny aspect of engineering. Your uptime numbers have lax requirements, your geographic distribution is severely limited, and your client counts (as in clients directly accessing your software) are orders of magnitude smaller than the Twitter/Uber you compare yourself to.
Operating a financial exchange it not trivial by any means, but to compare your challenges to a massive public-facing service is apples to oranges. There are likely more adversarial clients connected to twitter’s services at any given time than the peak concurrent connections to any given exchange.
> * If JPM goes down for 2 hours it will cause a congressional intervention?*
It will create a crisis meriting Congressional intervention. A hiccup in the tri-party repo market automatically sends, within twenty minutes, a large fraction of the American financial system into default within twelve hours from almost anywhere on the clock.
> That’s honestly really hard to believe and it borders in delusions of grandeur
Yep, repo and swaps are huge, our euro repo market alone is in around quarter of trillion a day..
People don’t understand what happens when some things go down, and how it would affect them.
Futures are critical to you being able to buy milk eggs and bread at the store day after day and not having the price fluctuating on you by 100’s of % or worse those items simply being unavailable because the price either spiked too high or completely crashed so the supply is nonexistent.
Same goes for many other commodities. A cock up in the futures market can and will send ripples through the entire commodities market which will have a direct impact on the world’s supply chain.
Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours or if the entire tri-party repo market fails? Those are not even remotely the same thing.
I would be shocked to learn that Exxon mobile will go bankrupt in half a day because one bank has an outage. It would speak volumes to how piss-poor financial networks are architected.
More realistically, JPM goes down and within an hour all of its large repo clients have already called BNY or whatever and are ready to go. Billion dollar businesses have contingency plans for headquarters being taken out by a bomb, you better believe they have a plan for their repo provider taking a shit.
> Lovely
Do you not see how ridiculous it is to suggest that the entire functioning US economy hinges on a single service run by a single company being available 24/7? If it were true, it would be a bigger terrorist target than the Whitehouse.
Apparently the entire US economy is dependent on the uptime of the work of a few engineers at one company. Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc. These engineers are literally more important than Congress to keep alive and working.
> Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours
Yes. Look up how tri-party repos work [1] and then look up the Fed’s commentary on their fragility.
One of the three legs is either JPMorgan or BoNY. These are short-term but critical loans. If one of those two fails or goes offline, which practically means failure since others can’t verify the other two legs’ liquidity, large swathes of the rates, credit, futures, regular repo and stock loan markets go offline which quite literally will lead to, maybe not Exxon, but the likes GE having payments failures. You can’t “hot swap” JPMorgan to BoNY after the contract has been initiated. It would be like saying it doesn’t matter if your brokerage fails, just call someone else. Sure, for your next trade that’s fine, but in the meantime the failed broker has your assets.
> Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc.
Friend is a senior IT guy at BoNY. They are regularly in touch with the Fed and FBI. It’s a known vulnerability, and there is constant scholarship and policy work on nationalising or reforming the tri-party repo market. But it’s never failed, and it’s profitable work for the two champions, so for now there are higher priorities for legislators.
Also, there are loads of these centralised pressure points in our system. In every modern financial system. Cede & Co., the Fedwire system, ADP, et cetera.
> It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.
A bunch of things:
- that wasn’t an outage, that was actual trades taking place at significantly decreases prices. Not even close to the same category. Those same exchanges involved in that crash literally go offline every evening and on holidays/weekends and nobody cares. In fact, if the exchanges had gone completely offline during that time period, it wouldn’t have even made the news because there wouldn’t have been a place for those trades to occur.
- it didn’t cause a trillion dollars to get wiped off the economy any more than my friend transferring a share of Apple to me for $0.50 causes a trillion dollars to get wiped off of the economy.
- nobody gives a shit about a “congressional hearing”. Do you realize how many congressional hearings tech companies have been pulled in front of now? A hearing != action and if you think otherwise, there has been significantly more “congressional action” over tech companies in the last couple of years than financial companies.
> If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.
Show me the public IP address of NYSE or whatever major financial exchange you want that I can connect to from any IP in the world. If exchanges were open to the world, they would get DDoSed out of existence in a heartbeat.
It never ceases to amaze me how easily the attitudes of finance folk so easily leak out. It seems performing a worst-case O(log N) lookup and spitting out the resulting 100 bytes over UDP counts as harder than Twitter, parent comment clearly has no idea how much work Twitter needs to perform for every tweet (URL previews much?)
Meanwhile, could have simply said you work for NYSE.
I don’t work for a stock exchange, stocks are not particularly high volume nor particularly that important for the market as many other instruments and asset classes.
Your comment shows you clearly have no idea how modern QDM and ODM markets work if you think that sending 100 bytes over UDP is all there is or even what we do, we don’t even use UDP...
An exchange like that by definition is quite focused on tight non functionals with slim functionals, aka it does a small number of things very well. Vs Uber which does a very wide set of things at a high level of consumer polish.
In my mental model of the two problems to solve, Uber is directly consumer facing to
Millions of humans and that alone creates so much work in engineering that doesn’t exist when you can craft an api with sub 50 ms latency consumed by thousands of other machines.. the humans create the complexity. Solving for their problems is why you need way more people.
Order matching, order execution, credit and limit controls, reporting, multiple APIs and interfaces for order entry, market data and many other things supporting everything from legacy FIX clients coming over leased lines to modern restful web APIs and SBE over websockets, identity management with highly complicated RBAC model for traders and firms (who can trade what on behalf of whom under which conditions), highly complex and robust logging especially for regulatory requirements, maintaining alignment with regulations across the globe...
And if you think normal end users are picky you haven’t met traders that still want to use a 25 year old keyboard with non-standard ASCII codes that’s barely recognized as an HID device on your new electron based trading frontend and if you can’t accommodate for that they will leave.
So you have millions of clients all of which are adversarial and you have to maintain market fairness and integrity to ensure no manipulation is possible with trillions of dollars on the line that can be wiped off in a heartbeat and that is before you start thinking about nation state actors who would love to be able to cause major economic damage to the US, Europe and other major western economies.
Yeah the complexity is insane, millions of users is surprising tho I thought it would more be an aggregation of larger agents.
I guess unicorns optimize for delivery capacity not cost
Lyft has only US and Canada operations. Vast majority of Uber employees will be in customer service and operations. It it no way comparable to GitHub and Nvidia
Lyft had minuscule global presence. Uber has enormous global presence, it is a top 2 player in India , not to mention it's presence in Europe and other parts of the world.
The sheer number of languages they have provide user support and staff for driver management in these two markets alone is enormous.
Does it have to be one or the other? This is an equally ignorant response. 21k is a lot, and to hand-wave any questioning of this as "ignorant" and "arrogant" is... those things. Lots of companies have bloat, and this is especially apparent if you've also worked somewhere that's competently managed. Recently someone was telling me about a shoe auction company employing hundreds of engineers (sorry, I forget the exact figure, want to say 300-600). I don't care how much traffic you're doing or how many shoes there are, that is a ridiculously unnecessary number.
This is sort-of hand-wavey. The truth is there are many companies with order-of-magnitude too-many employees (and software eats them). And there's some that aren't.
There's no way to know which one uber is without exactly breaking down those numbers a bit more. But call me unconvinced for one.
I don't have the experience to back my opinion, but my gut agrees.
Intuitively, it would seem to me that hiring past a certain threshold increases need to hire more engineers rather than quenching it — that is, the complexity resulting from changes made to the projects to keep such a high number of engineers productive approaches or even outstrips innate complexity and becomes unsustainable, demanding ever more engineers to keep it all afloat. Following this, a company that isn't keeping an eye out for net productive decreases as a result of hiring could end up with far more engineers than they actually need.
But I've always been an individual contributor, so my perspective may be limited.
I have a deep suspicion that this is mostly about risk management. With just 50 employees, what you have is essentially a board of directors that have enormous leverage over you.
> With just 50 employees, what you have is essentially a board of directors that have enormous leverage over you.
How true this is. I've seen what happens when you have a small number of very wealthy engineers in senior leadership roles in a small to midsize tech company that has gone public. They make it as difficult as possible to hire senior people from industry that's needed to fill critical skill gaps within the engineering org. Then all of the talented mid-level folks who have no internal mobility jump ship for Facebook.
Uber is complex, that’s for sure. It’s not a job for 10 people team. But they also suffer from not invented here syndrome and blow up amount of work needed.
That’s not unique to them. Lots of Silicon Valley companies have engineering teams seeded with folks who worked whole life at Google, Facebook, etc. They default to writing custom solutions for everything, because that’s what they were taught.
> They default to writing custom solutions for everything, because that’s what they were taught.
I can only speak regarding my experience at Google, but the company has been moving aggressively toward industry standards for software development, primarily at the language level.
I constantly run see internal libraries in C++ being updated to use standard C++ smart pointer types, and Java being updated to use types like java.lang.Instant instead of some internal implementation created years ago.
What does tend to be custom are the systems that get built, not only because of the huge scale of operation, but also because of Google-specific security and compliance requirements. Also, the development tools and continuous integration systems are also necessarily specific to Google (and are pretty amazing).
But even then, there is extensive use of certain externally originating systems - memcached is one that comes to mind.
> I can only speak regarding my experience at Google, but the company has been moving aggressively toward industry standards for software development, primarily at the language level.
Why now? What were technical reasons this wasn’t done before at Google?
Exactly. For example, like many companies, Google had smart pointer implementations in its C++ codebase almost from its inception, but they didn't appear in the C++ standard until C++11.
This is interesting detail, but it doesn't sound like it requires hundreds of engineers. Most of this seems like relatively common functionality (processing payments) with heavy regionalization. None of this is cutting edge work. Knowledge resources and experienced people would be widely available for all of it. It seems hard to understand why maintaining this system, even with the number of locations and users, would require more than 100 pure software engineers (thinking ~10 teams of 10, at most).
I'm never sure how much Parkinson's Law is at play with startup hiring in general. But I guess most engineers they hire provide value to the company at least equivalent to their salary.
That is a great explanation. It’s to Uber’s credit that their app seems so deceptively simple. That said all those special cases make me even more skeptical about them getting on the right side of profitability.
Every one of those localized special cases that Uber puts a team on is them defending a moat against localized competitors. For the upstart localized app, that country’s payment system, driver laws, airport rules etc more similar to the basic 5 screen app Uber seems to be.
Anytime Uber let’s it’s foot off the gas in a particular place one of those will show up, offer drivers a slightly better deal, and Uber will have to match - killing any margin.
What margin? They've never made a profit as far as I know. Maybe they'd make more money if they let these local competitors take their unprofitable business.
We shouldn't ignore the fact that Uber has been going for 11 years though. That's a lot of time to build all that functionality and iterate it to be good. A small team of competent devs could build everything in the comment you linked to in a little over a decade. Probably a lot less.
Build well? These things aren't built perfectly the first time, or anywhere close. You don't always know what you're building until you've iterated on it a few times. It's just ready to forget that you didn't know when you have the benefit of hindsight.
And build well and maintain? Not a chance. This should be obvious.
Maybe we have different ideas off what a small team is. To be, 3-4 devs is a small team. In context, I could say maybe ten developers, but I would consider that a large team. At that point I'd be preparing to split into multiple teams.
In not saying they needed 21,000, but it's likely we just mean completely different things when we talk about a small team. What does it mean to you?
I guess you missed the word "iterate" in my post. Good software doesn't spring in to existence just because a team has talent. It starts off OK if you're lucky and it evolves in to something good if the team can guide it properly.
- Assume 1000 for the apps.
- +1000 for backend.
- +1000 for infrastructure.
- +500 for SRE functions.
- +1500 G&A
That leaves us with 16,000 employees who are not in these categories. Supposing Uber operates in 50 major markets that require a team of 50 employees each (political fixers and such). That still leaves 13,500 to account for!
One explanation is that each vertical has this structure and employee count, which means that the 3 or so verticals (rideshare, eats, ??) together account for roughly 19500 employees.
I am not very familiar with Uber. I have only used them for taxis and that too a few years ago.
- Legal is generally part of G&A (General & Admin.)
- Support is one category that I hadn't thought about, but perhaps they are contractors.
- I can't imagine Uber having more than 50 or so design people, all in the 'apps' category.
What are the others in the 'tons' more category, not part of any of the categories I have mentioned? It may be a super mass market consumer service but part of the reason it's so highly valued is the efficiencies of scale it brings to these things.
Fifty design people? It's going to be at least an order of magnitude more. (To confirm, did a LinkedIn search for Product Designers at Uber. 521 results. Pretty close!)
I work at tech company you've probably heard of (consumer-focused, international, relatively large scale, though probably a couple orders of magnitude smaller than Uber). We have ~200 engineers (client, backend, ops/infra, data eng, ML, etc). We have ~15-20 designers. We're international but we do very little feature and/or design customization per market, beyond localization, unlike Uber, which does customization even for different cities in the same state, let alone separate countries!
Good heavens! What are 500 design people doing at Uber? Is this a case of managerial empire-building? I was at a small company of a comparable size to yours where the product was a mobile app (Apple and Android). It had fewer than ten (roughly 6) designers. This app was reasonably complicated (but, reading an account of Uber's struggles with controlling the size of it's app, nowhere close to Uber's apps' complexity) and was very important in the general scheme of things.
Companies like Uber who dance with the law, have their executives arrested abroad like in France or Canada, or implement schemes to evade local officials building cases against them, must have a huge legal (and costly) department.
Yeah and it even gets better that they now have a direct connection to the US (Vice) President. Just, what could go wrong with that and mob-like anticompetitive behavior...
There are several arguments for why companies need so many employees, but I don't buy any of them. Scale carries with it complexity, but only so far. WhatsApp had < 100 employees when it was bought for $19bn, with hundreds of millions of users. It doesn't get much bigger than that. Sure, Uber is more complex than a chat app, but 200x more complex? I don't think so.
even a few things like Country specific payment, fraud detection, routing, identity verification, price calculation make it way way complex don't you think?
I don't think VC-funded companies usually hire based on concrete needs, it's more like "we're growing at X rate so we can budget Y for more heads" and then management back-solves for where they can best deploy resources accordingly.
If you go drive for uber, there is often a support center called a 'greenlight hub' and it's kind of like an Uber DMV / support center in your local city. Are those people employees? They might be.
There are approx 700 cities for uber according to a random google search, lets say 10 people work at each center. That's 7000 employees alone. Then you have all the support staff, people who have to manage cities and so on. Uber is not a pure software business like twitter or FB is, they need a lot of 'retail' staff like apple does.
"Exits" almost always destroy the original value created in a company. I get offers all the time to buy my company. Almost all of them come from men (and yes, they all have been men so far) with very little experience in my industry. If I sold to them, I would make a ton of money... But it would almost certainly destroy a lot of value in the company itself. My clients would get worse service, and my employees would not have the promised growth and benefits. And I think I would lose the meaning and purpose my company gives me.
Avoid VC's and outside investors like the plague of you can. Their desire for an exit will eventually force you into one.
> Most of the executive team at Postmates, including Bastian Lehmann, the founder and chief executive of the popular food delivery app, will leave the company
Especially after reading that, this looks like a standard post-acquisiton redundancy reduction. I'm sure it still sucks for these people, but they probably also got retention bonuses after the acquisition, severance packages from Uber, and saw this coming from a mile away.
95 comments
[ 2.6 ms ] story [ 186 ms ] threadUnbelievable, I can’t even think what type of work they do.
WhatsApp with 50 employees is an anomaly. In reality, companies need lots more employees just to get by. Not all industries are equal when it comes to responsibilities and programmers are the most ignorant but arrogant class, thinking that all you need is software. Spolsky covered this 2 decades ago in his “Developer Abstraction Layer” article.
Also MoviePass was at a good start by selling tickets at a loss, what killed them was that they couldn’t leg into the profit generating phase of their strategy and burnt out.
I haven't heard of NVIDIA owning factories.
Any given project might not require large numbers people, the headcount driver might the process.
Where you have control, would recommend rethinking the model to look more like Whatsapp than to dismiss as an anomaly.
I'm sure it takes more developers to build and support Uber than you might initially guess, but I'm also sure that Uber just has more employees than they need "because they're successful".
But the vast majority of Uber's employees are not software engineers. Running a physical service in ~70 countries is /hard/ and requires a lot of people. Support and operations eat up a lot of the headcount, and can't be scaled like software. Similarly you need a lot of accountants and lawyers to operate globally, along with all of the infrastructure to support a global workforce.
[Disclaimer: I work for Lyft]
I suspect they might be a few percent overstaffed, but I don't think that hurts a company focusing on growth over all else, and I don't think it's even all that uncommon for such companies. It wouldn't surprise me to see a round or two of layoffs once that growth tops out, but it would surprise me if they added up to more than 10% of headcount.
Being number 2 often just means copying the number 1 player in a more capital efficient manner, product descriptions become "X did it this way, customers want it this way". In a winner takes all market, being the first mover matters a lot.
Edit:
Source: https://www.ft.com/content/9d9624a4-8341-11dd-907e-000077b07...
We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.
We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.
We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
21K for Twitter... Lol...
What products are exclusively traded on your exchange? If NASDAQ or NYSE goes down for half a day it’s just an article on WSJ at best. They shut down every night and for entire days on holidays. What financial exchange truly has a meaningful impact when it goes down?
A more critical and delicate financial artery is the tri-party repo marker. If BoNY Mellon or JPMorgan’s repo desks went offline, it would almost certainly shut down large swathes of the American economy, prompt a credit crisis and merit urgent Federal Reserve and possibly Congressional intervention.
My point is that your strange Internet dick measuring contest of micros to match up an order book is only one tiny aspect of engineering. Your uptime numbers have lax requirements, your geographic distribution is severely limited, and your client counts (as in clients directly accessing your software) are orders of magnitude smaller than the Twitter/Uber you compare yourself to.
Operating a financial exchange it not trivial by any means, but to compare your challenges to a massive public-facing service is apples to oranges. There are likely more adversarial clients connected to twitter’s services at any given time than the peak concurrent connections to any given exchange.
It will create a crisis meriting Congressional intervention. A hiccup in the tri-party repo market automatically sends, within twenty minutes, a large fraction of the American financial system into default within twelve hours from almost anywhere on the clock.
> That’s honestly really hard to believe and it borders in delusions of grandeur
Lovely.
People don’t understand what happens when some things go down, and how it would affect them.
Futures are critical to you being able to buy milk eggs and bread at the store day after day and not having the price fluctuating on you by 100’s of % or worse those items simply being unavailable because the price either spiked too high or completely crashed so the supply is nonexistent.
Same goes for many other commodities. A cock up in the futures market can and will send ripples through the entire commodities market which will have a direct impact on the world’s supply chain.
I would be shocked to learn that Exxon mobile will go bankrupt in half a day because one bank has an outage. It would speak volumes to how piss-poor financial networks are architected.
More realistically, JPM goes down and within an hour all of its large repo clients have already called BNY or whatever and are ready to go. Billion dollar businesses have contingency plans for headquarters being taken out by a bomb, you better believe they have a plan for their repo provider taking a shit.
> Lovely
Do you not see how ridiculous it is to suggest that the entire functioning US economy hinges on a single service run by a single company being available 24/7? If it were true, it would be a bigger terrorist target than the Whitehouse.
Apparently the entire US economy is dependent on the uptime of the work of a few engineers at one company. Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc. These engineers are literally more important than Congress to keep alive and working.
Yes. Look up how tri-party repos work [1] and then look up the Fed’s commentary on their fragility.
One of the three legs is either JPMorgan or BoNY. These are short-term but critical loans. If one of those two fails or goes offline, which practically means failure since others can’t verify the other two legs’ liquidity, large swathes of the rates, credit, futures, regular repo and stock loan markets go offline which quite literally will lead to, maybe not Exxon, but the likes GE having payments failures. You can’t “hot swap” JPMorgan to BoNY after the contract has been initiated. It would be like saying it doesn’t matter if your brokerage fails, just call someone else. Sure, for your next trade that’s fine, but in the meantime the failed broker has your assets.
> Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc.
Friend is a senior IT guy at BoNY. They are regularly in touch with the Fed and FBI. It’s a known vulnerability, and there is constant scholarship and policy work on nationalising or reforming the tri-party repo market. But it’s never failed, and it’s profitable work for the two champions, so for now there are higher priorities for legislators.
Also, there are loads of these centralised pressure points in our system. In every modern financial system. Cede & Co., the Fedwire system, ADP, et cetera.
[1] https://info.bnymellon.com/rs/651-GHF-471/images/BNY_Tripart...
It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.
If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.
A bunch of things:
- that wasn’t an outage, that was actual trades taking place at significantly decreases prices. Not even close to the same category. Those same exchanges involved in that crash literally go offline every evening and on holidays/weekends and nobody cares. In fact, if the exchanges had gone completely offline during that time period, it wouldn’t have even made the news because there wouldn’t have been a place for those trades to occur.
- it didn’t cause a trillion dollars to get wiped off the economy any more than my friend transferring a share of Apple to me for $0.50 causes a trillion dollars to get wiped off of the economy.
- nobody gives a shit about a “congressional hearing”. Do you realize how many congressional hearings tech companies have been pulled in front of now? A hearing != action and if you think otherwise, there has been significantly more “congressional action” over tech companies in the last couple of years than financial companies.
> If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.
Show me the public IP address of NYSE or whatever major financial exchange you want that I can connect to from any IP in the world. If exchanges were open to the world, they would get DDoSed out of existence in a heartbeat.
Meanwhile, could have simply said you work for NYSE.
PP’s NDA probably doesn’t allow that.
Your comment shows you clearly have no idea how modern QDM and ODM markets work if you think that sending 100 bytes over UDP is all there is or even what we do, we don’t even use UDP...
In my mental model of the two problems to solve, Uber is directly consumer facing to Millions of humans and that alone creates so much work in engineering that doesn’t exist when you can craft an api with sub 50 ms latency consumed by thousands of other machines.. the humans create the complexity. Solving for their problems is why you need way more people.
And if you think normal end users are picky you haven’t met traders that still want to use a 25 year old keyboard with non-standard ASCII codes that’s barely recognized as an HID device on your new electron based trading frontend and if you can’t accommodate for that they will leave.
So you have millions of clients all of which are adversarial and you have to maintain market fairness and integrity to ensure no manipulation is possible with trillions of dollars on the line that can be wiped off in a heartbeat and that is before you start thinking about nation state actors who would love to be able to cause major economic damage to the US, Europe and other major western economies.
The sheer number of languages they have provide user support and staff for driver management in these two markets alone is enormous.
There's no way to know which one uber is without exactly breaking down those numbers a bit more. But call me unconvinced for one.
Intuitively, it would seem to me that hiring past a certain threshold increases need to hire more engineers rather than quenching it — that is, the complexity resulting from changes made to the projects to keep such a high number of engineers productive approaches or even outstrips innate complexity and becomes unsustainable, demanding ever more engineers to keep it all afloat. Following this, a company that isn't keeping an eye out for net productive decreases as a result of hiring could end up with far more engineers than they actually need.
But I've always been an individual contributor, so my perspective may be limited.
How true this is. I've seen what happens when you have a small number of very wealthy engineers in senior leadership roles in a small to midsize tech company that has gone public. They make it as difficult as possible to hire senior people from industry that's needed to fill critical skill gaps within the engineering org. Then all of the talented mid-level folks who have no internal mobility jump ship for Facebook.
Here's a thread [0] regarding the complexity of Uber's iOS app.
Dan Luu made a far better case [1][2] for this than I could. Worth the read.
0: https://threadreaderapp.com/thread/1336890442768547845.html
1: http://danluu.com/sounds-easy/
2: https://twitter.com/danluu/status/782974159156502534?lang=en
I don't object to lots of employees in principle, but coupled with losing money every single quarter, it's a red flag for me.
That’s not unique to them. Lots of Silicon Valley companies have engineering teams seeded with folks who worked whole life at Google, Facebook, etc. They default to writing custom solutions for everything, because that’s what they were taught.
I can only speak regarding my experience at Google, but the company has been moving aggressively toward industry standards for software development, primarily at the language level.
I constantly run see internal libraries in C++ being updated to use standard C++ smart pointer types, and Java being updated to use types like java.lang.Instant instead of some internal implementation created years ago.
What does tend to be custom are the systems that get built, not only because of the huge scale of operation, but also because of Google-specific security and compliance requirements. Also, the development tools and continuous integration systems are also necessarily specific to Google (and are pretty amazing).
But even then, there is extensive use of certain externally originating systems - memcached is one that comes to mind.
Why now? What were technical reasons this wasn’t done before at Google?
I'm never sure how much Parkinson's Law is at play with startup hiring in general. But I guess most engineers they hire provide value to the company at least equivalent to their salary.
Every one of those localized special cases that Uber puts a team on is them defending a moat against localized competitors. For the upstart localized app, that country’s payment system, driver laws, airport rules etc more similar to the basic 5 screen app Uber seems to be.
Anytime Uber let’s it’s foot off the gas in a particular place one of those will show up, offer drivers a slightly better deal, and Uber will have to match - killing any margin.
Build well? These things aren't built perfectly the first time, or anywhere close. You don't always know what you're building until you've iterated on it a few times. It's just ready to forget that you didn't know when you have the benefit of hindsight.
And build well and maintain? Not a chance. This should be obvious.
Maybe we have different ideas off what a small team is. To be, 3-4 devs is a small team. In context, I could say maybe ten developers, but I would consider that a large team. At that point I'd be preparing to split into multiple teams.
In not saying they needed 21,000, but it's likely we just mean completely different things when we talk about a small team. What does it mean to you?
That leaves us with 16,000 employees who are not in these categories. Supposing Uber operates in 50 major markets that require a team of 50 employees each (political fixers and such). That still leaves 13,500 to account for!
One explanation is that each vertical has this structure and employee count, which means that the 3 or so verticals (rideshare, eats, ??) together account for roughly 19500 employees.
I am not very familiar with Uber. I have only used them for taxis and that too a few years ago.
What are the others in the 'tons' more category, not part of any of the categories I have mentioned? It may be a super mass market consumer service but part of the reason it's so highly valued is the efficiencies of scale it brings to these things.
I work at tech company you've probably heard of (consumer-focused, international, relatively large scale, though probably a couple orders of magnitude smaller than Uber). We have ~200 engineers (client, backend, ops/infra, data eng, ML, etc). We have ~15-20 designers. We're international but we do very little feature and/or design customization per market, beyond localization, unlike Uber, which does customization even for different cities in the same state, let alone separate countries!
It's an accidental complexity pump.
https://www.nytimes.com/2017/03/03/technology/uber-greyball-...
https://techcrunch.com/2015/06/29/uber-france-leaders-arrest...
If I attempted to do that same in my home country without a huge VC backing thus very good lawyers, I'd be shut down in no time and sent to jail.
They have some tech sure, but primarily they are a transportation company. They act like a tech company to justify the insane valuation.
Your question is like to asking why ups needs 480,000 employees when whatsapp needs 100 for similar number of users.
There are approx 700 cities for uber according to a random google search, lets say 10 people work at each center. That's 7000 employees alone. Then you have all the support staff, people who have to manage cities and so on. Uber is not a pure software business like twitter or FB is, they need a lot of 'retail' staff like apple does.
Avoid VC's and outside investors like the plague of you can. Their desire for an exit will eventually force you into one.
Especially after reading that, this looks like a standard post-acquisiton redundancy reduction. I'm sure it still sucks for these people, but they probably also got retention bonuses after the acquisition, severance packages from Uber, and saw this coming from a mile away.