Wow... Being down once - OK, it was record-high volume, servers were strained. Twice - well, it was only a day after the first time and it takes time to fix these things. But a third time, nearly a week after the first incident (and once again on a volatile day where users stand to gain/lose a lot of money)? Really hard to think of any acceptable justification for that.
The simple answer is that the reason the system failed the first time wasn't easy to fix in the short term. Either they need to acquire more hardware resources or redesign some system so that it scales better. Not everything can be fixed overnight. Take a look at reddit - that's a website that collapsed repeatedly for months and months and months. Sometimes it's not something you can just 'fix'.
If they KNEW it was broken and were working on a fix, then they should of shut down the service. I'm sorry but this is simply unacceptable. I don't use Robinhood, but when you are handling very time sensitive contracts that can make or lose people a lot of money in a very short period of time if the service is not stable it should not of been in operation until it was fixed. I can't believe HN is making excuses for this.
If they do that, then they're definitely harming any customers who have a position they need to get out of, as opposed to only potentially harming them.
If you believe that the platform is so broken that it shouldn't be used (and I'm not arguing against that, btw) then a more reasonable solution would be to only allow position-reducing trades. That would also reduce overall trading volume on their platform.
Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.
Provision the resources and take the financial hit in costs.
If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent.
Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so.
A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so.
> Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.
> The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so.
Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.
All brokers were able to do it, they just didn't because commissions were easy profit and there was no competition.
$0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.
If you horizontally scale all the way, you just take on different issues like CAP theorem trade-offs. Financial book-keeping is a hard thing to scale and all it takes is a single component that cannot, like the transactional bedrock of your system.
I don't think getting rid of the people who know the system best will help anything. Some new guy who comes in with no clue of what happened or any of the codebase is not going to do a better job. It's just going to reduce long term productivity and increase risk.
I'd argue that the consequences of reddit not being available for a certain amount of time are absolutely not comparable with a financial services' firms infrastructure.
Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money.
Comparing it with a site like reddit is, frankly, a bit disengineous.
Consequences, no. Infrastructure failures, absolutely. What's being hosted isn't as important when it's broken and you're struggling to fix it. Sure, when it's a financial service's infrastructure, you're worried about other peoples money, but it's still something that's broken and may take time to fix.
The load testing you need to do for trading is considerably higher than any chat forum to go live .
Your customers are paying for speed and performance .
In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market
I don't think that argument really holds up when Robinhood is the subject. I mean they still don't even have phone support or same day guarantees covering them responding to your email.
Robinhood is nothing like Reddit. It has a much bigger valuation, more resources, strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system.
The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.
> strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system
Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.
I know it's not the same regulations but I'm pretty sure the next FDA auditor would have a heart attack if we slipped "rewrote the whole product" into one of our patch notes at work.
Being in a regulated environment with lots of peoples money should make it harder and take longer to redesign and implement anything. I don’t see a complete redesign of a non-trivial part of a trading platform happening in a week.
Your literal quote was "This should be fixed within days even if it means redesigning the entire system". And my position is, being a regulated financial service, it's probably not reasonable to redesign (and build, test, deploy etc) a system in days.
"redesigning the entire system" was an exaggeration. They have a serious problem if that's really what they need to do.
Rolling out fixes fast, even if they require intensive changes, is completely reasonable and expected in many industries though. They should have the talent and procedures to get it done. This isn't some startup web app, it's a multi-billion dollar broker managing millions in client funds. It's bordering on incompetent to have 3 outages in 2 weeks.
I understand the risks need to be weighed against restoring service for their users who might be losing money and avoiding regulatory fines.
Changing a single line can introduce a different bug. Use proper QA and testing to catch as many as possible as with any development.
My emphasis is on getting things fixed quickly. They need to do whatever it takes to get systems online asap. Not sure what's so controversial about that.
Well to be honest we don't know how badly things are broken only they do. But the concept of moving fast while doing things perfectly is the holy grail of software development and not as easily achieved as you seem to make it appear.
I only said it needed to be done fast, regardless of how much work. Of course it's not easy.
I'm surprised by all the misinterpretation in this thread. Seems like it reflects the laid-back West Coast/SV attitude that isn't a good fit for high pressure time-sensitive work in other industries.
> needed to be done fast, regardless of how much work
This is the part you don't understand. There is a difference between digging ten one-foot deep holes vs one ten-foot deep hole. People need time to plan how to coordinate and then get on the same page so that everyone can work at their own pace. That is the part that is not parallelizable and is the rate-determining step.
The context is all lost here. Plenty of other companies and industries have emergency action and disaster recovery. People don't work at their own pace, they work to the deadline with solid procedures. They can fix and replace entire components to restore service ASAP since because that's the priority.
If this sounds unfamiliar or onerous then it's because you and others might have never experienced teams that do this. Robinhood is clearly lacking this experience and disaster planning.
You seem to think (or at least imply) that hard things can be done fast if only you work hard enough at it, that it's just a matter of trying. This is just not true. Not sure how else your posts should be interpreted. I've spent days with a team (yes a competent team) just tracking down a bug, let alone fix it (although usually when it's tracked down it's relatively quick to fix). If this issue involves multiple systems in a highly complex environment, then it very well could take a while to address fully, no matter how hard they work at it.
Because that's usually the case. Crunch time, disaster recovery, and emergency fixes are common in every sector from video games to aerospace. If you can't fix then switch to a secondary, or rebuild from backup, or throttle users, or process manually, or do anything other than be completely down.
RH wasn't prepared with any contingency. They should have a resolution for their users - even if they can't find or fix the original cause. That's the failure I'm talking about.
See the 2 other users in this thread that describe similar high-pressure situations.
Depends on what the issue is. Could be something that can be quickly fixed or not. Although going by the lack of a resolution I’m assuming it’s not.
Reality is, without knowing more about what’s causing this it’s impossible for either of us to say. If there is indeed some fundamental bottleneck that was previously not known, then I certainly won’t be surprised if it takes a while to sort out.
Now you can say they should’ve load tested, capacity planned etc etc. But we are where we are. Still can’t go back in time to turn this into a quickly fixable problem if it’s currently not.
Edit: also pretty disappointed that we don’t know more about the root cause. As an user I’d want to know what the issue was and what they are planning to do to about it to evaluate if I should trust them going forward.
While I don't agree with OP about replacing an entire system overnight. I do remember a friend who worked on a trading floor, and when they had bugs, their manager would say "the traders are taking a 30 minute lunch, you have that long to fix it" with the clear implication that they'd be fired if they didn't.
So I'm not sure if I know that the state of the art for trading platforms is as rock solid as everyone is implying, and Robinhood seems to be way far off from whatever gold standards there are, see infinite leverage bug. So I don't think it's crazy that they move quickly to fix it.
I'd never be able to stomach the pressure, and I wouldn't wish it on others, but it doesn't seem crazy.
The last 20 years I have been working in heavily regulated environments; in cock-ups like this were your Retail clients got royally screwed (for a second time in such a short period), on a repeat issue, the hammer will fall HARD on their heads. Anyone wanted to sell to minimize losses has been severely damaged.
There are many ways to resolve these issues FAST. The easy way is to throw money at this. Go tomorrow (literally) and bypass all procurement controls and go to a mega big provider and scale this asap. Any financial services with a half decent IT has done the paper exercise to this scenario (at least for the purposes of BCP/DRP). The slow/better/mature way may be too slow, especially with the current market conditions.
The RH folks will definitely get a visit from SEC, their external auditor, and their external auditor will get a visit from SEC.(their auditor will be in the deepest of shits)(how come they failed to spot such a going concern issue?)(what the hell were they looking for on their audits?)(did they only send juniors over there?)
I feel sorry for the retail traders that got knocked down. I think that anyone locked in buying at 27-28k (US30) should wait 6 months to breakeven and after the US elections (irrespective of the winner) there will probably be a rally.
They aren't that fantastic either. They had issues today as well, I couldn't exit out of spreads, so I had to liquidate everything and that was failing too. It was a hot mess.
Edit: I didn't actually want to liquidate all my holdings but it was better than staying in and risking everything else.
The market makers and exchanges stop executing orders because a series of circuit-breakers fired (set by the regulating organizations).
Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.
What sets financial services apart when they fail is the extreme legibility of the losses. One can argue that if say Reddit is down, they have losses, but measuring the impact in terms of dollars lost is difficult.
It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.
I get the impression that the people running Robinhood actually don't understand how difficult it is to implement what they want to offer, both on the reliability side (outages) and on the logical side (infinite margin).
This is pretty interesting for capacity planning. I worked at a minor ecommerce company when Black Friday was still a big deal. In the month leading up to it, we went into code freeze and made sure all our systems can handle added load.
What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.
This is one of the problems that "the cloud" is supposed to solve. No need to keep infra up for that unicorn event, just to ensure that you can scale properly when it does happen.
Until you hit some kind of aggregation server that it's not trivial to scale out properly unless you take a completely different approach or that you may scale vertically up (but eventually you will hit the ceiling).
Matt Levine had some interesting (and funny!) insights the last time [0]
> It is well known that one of the best services a retail broker can provide is not answering the phones during a crash. The market is down, the customers panic, their timing is terrible, they want to sell at the bottom, they call you up to say “sell everything,” you say “we’re sorry all our representatives are assisting other customers, your call is important to us,” they hang up and get distracted, the market rallies, they forget about selling, you have saved them a fortune, good work.
Seemingly this is the strategy of many Bitcoin exchanges as well, as every time Bitcoin price is going down, mysteriously Coinbase is having issues with their systems.
I just happened to have watched The Big Short [0] yesterday.
In the movie (based on real events) the same thing happened: during the housing crash suddenly every bank had 'technical issues' when people tried to sell their stocks.
Indeed ! If I remember correctly though, they had difficulties only until they sold all their crap (hence lying to their clients by selling them crap as gold). Once they were freed from their toxic assets, they "allowed" everyone to sell. What a world we live in.
Disclaimer: I'm not in finance, don't know much about it, but it's what I understood while watching the movie ;)
That's the point of cryptocurrency though: a financial world unencumbered by those annoying regulations that protect the buyer. No takebacks.
Reality is the regulations on the financial system are a feature. Being able to have charges reversed if I get my card skimmed and someone makes a bunch of charges is a feature. Being able to get my money back if the seller ships me broken goods (or nothing at all) is a feature.
Having regulations on banks front-running their customer's transactions is a feature (one that Robinhood unfortunately gets around somehow). Having regulation on securities not being scams is a feature.
> Reality is the regulations on the financial system are a feature.
As with anything, the answer is sometimes.
You want those rules for your retirement account, because if someone can steal a six figure sum from you instantly with no takebacks, that is bad.
But the overhead of that system is not always required, and it's the major reason that we don't have e.g. micropayments. The overhead of allowing transactions to be reversed eats the entire transaction amount for very small transactions.
So it would be good to have a system where you could, for example, transfer $50 into it from your bank account, wait the month or so until the transaction can no longer be reversed, and then transfer it from there a few pennies at a time with insignificant transaction costs because the small transactions are instantaneous, anonymous and can't be reversed. Which isn't nearly as problematic because even if you get scammed, your loss is limited to the $50 you put in. Instead of getting wiped out, you pay a reasonable cost for a personal lesson on security and trust.
And there is no reason we couldn't have both. Then you keep the bulk of your wealth in the inefficient safe system and some petty cash in the one with lower transaction costs and get the best of both worlds. But the existing rules don't allow that, which is bad.
That's somewhat correct. But it wasn't that they didn't allow those people to sell. In the film they were dealing with specialty derivative products that weren't traded by your average person. The derivative products in the film were sort of like bond pricing where price discovery isn't obvious and determining a fair price is a bit subjective since the volume was very low. So when people like Michael Burry (Bale's character) called to see what kind of price they could get, the banks would give him a bullshit, obnoxiously low offer price they'd be willing to give him because correctly valuing these derivatives would bankrupt the banks. So they held them on the books with fake valuations to stay afloat while they reduced their risk. Then, after doing all that they could fairly price them, which is when Burry makes a metric shit ton of money basically overnight - because he is finally able to get a fair price.
For example, if we took a simpler example of something more people are familiar with like options...let's say you buy a put at a strike of 100. The stock is trading right around 100. The market tanks like it has been doing and the stock goes down to 80. That's great for you since you had the put. You technically have the right to buy the shares at the market and shove them onto someone and force them to pay you 100. Most of the time, since we're dealing with derivatives, they just trade the value of the option itself and skip dealing with the underlying security because it takes a lot of capital (100 shares * 80 bucks each in this case). So the value of the option (the derivative) should be around $20 as you approach expiration and the premium fades off. If the market maker for the options has way too much exposure and wrote naked puts (meaning they didn't have the underlying security) they took on a ton of risk and basically got fucked. In this scenario it's like Burry calling and finding out his options are worth like $1.50 instead of $20 simply because they are thinly traded and therefore can't get a good price by anyone. This actually happens on some options - you'll see a huge spread between the bid and ask on some options so it's not like it doesn't happen even for this type of retail-friendly derivative.
This is a lot different though. Banks were the actual dealers of the securities in the big short- you were actually buying and selling to the bank itself. Robinhood and stock brokers, you aren't buying and selling to them- their primary function is to just give you access to the markets where market makers are participating on the other side of the deal. Robinhood does not participate in market making, they purely give you access to them.
In the Big Short case, the banks were acting as market makers, or more technically in this case dealers- this was more akin to going to a used car lot. You read a lot of reports saying that the new Fords were flying off the lots for outrageous prices, and to people with dubious ability to pay for the sticker prices they did because they got loans they shouldn't have. You go to the dealer and say I want to sell these cars short and buy them back later. The dealer presumably thinks you are an idiot, or wants to hedge some risk (the analogy is breaking down here), but says ok sure, thinking he is going to sell them for even more in 6 months. 6 months later comes- you were right- there is a flood of Fords on the market at half the price because so many people had them repossessed or are desperately trying to get out of these loans that are killing them. You go to the dealer and say "Ok bud, I'd like to buy these cars back at half price..." and they say "Nah, these are still worth 98% of what you sold for... how about that price?" and you are kind of stuck. You bought specific used cars that are kind of but not really entirely fungible. You can't just go down the street and buy those cars and replace them. You have to hope that they feel the pressure to lower those prices because they start feeling the squeeze for cash, or a regulatory agency comes in and puts the pressure on. They can kind of live in la-la land and avoid the reality of the situation as long as they have no requirement to sell.
Eventually though, they blinked and once one bank started taking write downs, all banks did, and once that became acceptable, they all followed suit- and they also needed the cash at that point.
Anyway- now back to stocks/options- these have well known discoverable prices and you can sell them on open markets where the brokers you connect to might be on the other end of the deal but its really unlikely. A broker/dealer like Robinhood or Schwab or Interactive Brokers should theoretically have an entirely flat position at the end of the day.
TL,DR: Robinhood didn't go down because it doesn't want you to sell your stocks.
Same here. I don't remember the details, but I had BTC (transferred from local wallet) stuck in some unexplained "pending" state for more than a week and Coinbase support was nowhere to be found. Thankfully, it was eventually resolved and that was the end of my BTC adventure.
One of the problems with Bitcoin is that there is a finite and fairly low limit of transactions that can take place per block. You have to get the bitcoin onto an exchange before you can trade them (and it is not recommended that you leave them there due to risk of scams/hacking) and during a time of crisis everybody else is trying to move their coins to the exchange too.
This means that transaction fees spike to 10-100x their normal rate during those crises, or else your transaction just sits and takes a week to get confirmed (if it ever does at all - eventually it will just time out after a few weeks).
The reason is quite simple: they are using Ruby on Rails internally for many parts of the system.
As much as I love Rails, it's extremely hard to scale an application written in it to handle millions of users. All the Active Record transactions have to be rewritten as database transactions if you want to make sure that the databas doesn't lock the tables for a long time.
Both Twitter and Groupon scaled to many millions of users using RoR. I don't remember it being especially more difficult than scaling any other kind of system to millions of users.
> The reason is quite simple: they are using Ruby on Rails internally for many parts of the system.
The choice of a language or framework is only one factor on the success of failure of a system.
You can write a broker in brainfuck if you are so inclined. You can make it rock solid. Your pick may require more or less engineering resources, but that's all it does.
You mention ActiveRecord. That too is premature optimization. Crafting optimized SQL queries will not save a badly architected app. They might even be worse if whoever is optimizing them doesn't fully understand what they are doing.
> if you want to make sure that the databas doesn't lock the tables for a long time.
You didn't mention which database you are using. You didn't say anything about its schema. This is yet another generalization. Not all databases will "lock the tables", no matter how badly queries are written. And, if they do, optimize that.
Usually, only a handful of services are critical. Microservices are all the rage, optimize those to death.
There's logic to this... Most exchanges have most of their funds offline, and only a small portion on their servers (to reduce losses in case of a hack).
When volumes go up, the servers run out of money, and everything grinds to a halt till humans intervene.
The humans typically want to do a lot of checks they aren't being tricked into refilling the online wallet too. Those checks are usually manual and take time.
This was true of stocks, but if you are holding options or other derivatives which are time-dependent then a market being closed is worse case scenario. RH is largely known for their options trading after all.
Not being able to exit these high-risk positions won't just bring your account to zero, but potentially put you in the six-figure negative if you are unlucky.
Buying the dip always feels intelligent in a bull market. In a bear market, it's not so wise.
The future is unknown to all of us. In some cases, looks are not deceiving. When the market appears to be on the cusp of freefall, it can in fact accelerate downward.
That's a slogan of pro traders who have three problems:
(1) They trade on margin, so huge drops are much more dangerous to them
(2) Their boss sees a Profit & Loss for the trading day that evening, if they don't like what they see they might give their portfolio to somebody else.
(3) Customers are calling the sales desk and they want to sell; hopefully sales can slow them down, but you may need to sell to pay for redemptions. Many customers may need to sell to rebalance their portfolios, get liquidity, etc. This is a mechanism which can carry instability from one market into another one which would otherwise be doing just fine.
When my wife and I bought our first house in 2006, the prevailing wisdom was that real estate was an incredibly safe (if often conservative) investment. Then 2008 happened. From that experience, I've decided that I will never invest in a market I don't understand.
That's the beauty of long-haul investing, though. Two years is nothing in terms of an investment strategy, and real estate, along with pretty much everything else, has rebounded since then.
There are a handful of hyper-geniuses that can make gobs of money by actively trading. For the rest of us, there's buy and forget.
The lucky bastards are the ones selling get rich quick schemes. The hyper-geniuses are the ones keeping their mouth shut and managing the Harvard Endowment.
You mean they are genius in the sense that they get paid to manage other people's money?
Because the Harvard endowment, like most all other funds out there, generally fails to beat the market and has had down years even while the market was up. This is probably due to lower risk tolerance, but it doesn't really show that hyper geniuses are running it either.
We bought our house in 2009. It was kinda scary--there was no telling when the recovery would come, and I was working a temp job that ended the day we signed papers. But we had been saving for a downpayment for several years--by 2005 or so it was clear that the Great Bubble was going to end very badly, so we just hunkered down in our crummy tenement apartment and waited it out.
Recovered in what way? I know a couple that just recently had the price of their home return to the price they paid for it over 10 years ago. With inflation taken into account, the house is still worth less than what the couple got it for.
It obviously depends on where you are (hence _most_ western markets) but in most parts of the UK and Ireland (and indeed many places in the US) if you bought at the _peak_ of 2007, mortgaged to the hilt, if you sold now, you would break even on inflation.
You're also forgetting that they had somewhere to live for a decade for the cost of inflation of the house, which is significantly cheaper than normal rental prices in many places.
To come out even after a decade, after housing costs, seems a pretty "safe" investment to me. Certainly not as lucrative as having invested in an index tracker, but it beats the 0% interest my bank pays on my current account balance.
There's also the saying though of "buy when there's blood in the streets, especially if it's your blood". All got to go up sometime. Just how much capital do you have to wait out the drop?
We're not near any fundamental limits on growth. They exist, but the economy can get a lot bigger before it hits them. I have heard some really daft arguments about it ("the observable universe is finite!") and some less daft arguments ("resource shortages are more likely than asteroid mining within our lifetimes") but all of these predictions are for things a long time in the future.
Yes, it does. All of the processes that have driven growth in the past are still working, and they will until they reach their limits. That's what it means for them to have limits. Why would people stop engaging in economic activity while it's still possible to do so?
I view it as such: do you believe we have achieved peak efficiency with regards to production of goods and providing services? If not, there’s room for the economy to grow.
I am not an economist though, this is just my layman’s view.
The actual question is whether the market thinks that future efficiency gains are on the way or not. If the market does, then that efficiency is already priced in and you can't make any money off of it.
What you actually have to figure out is: is there any unexpected future efficiency gains that the market doesn't expect. You have to know something that the market does not.
> The actual question is whether the market thinks that future efficiency gains are on the way or not. If the market does, then that efficiency is already priced in and you can't make any money off of it.
Isn't this ignoring opportunity cost, uncertainty, etc.? If the expectation of future gains is priced in, wouldn't treasury bonds sell for face value + remaining interest on the secondary market?
This isn't quite true. I get what you're saying, but from a technical perspective, it's not accurate. The markets also go up based on the total money supply in the system. If the government takes on a lot of debt, eventually a portion of that makes its way to the stock market, among other places.
You don't actually need to improve efficiency/productivity if the government has tools like the discount rate, open market operations, the ability to alter margin requirements on derivatives, fed funds rate, as well as programs and policies like MMIF, TAF, CPPF, ABCP, TALF, ZIRP, to manipulate the money supply and the velocity/flow rates of money.
That was coined for investment capital that comes with a cost. E.g. (in its simplest) borrowing money to trade with, and then having to make regular payments to maintain your positions.
Assuming you have no pressing needs for your invested capital (e.g. most retail investors), the market cannot actually stay irrational longer than you can remain solvent.
Time-based options have a fixed cost which you pay up front. So again, there is no risk of insolvency because you've already paid up front and capped your losses.
Yes, nobody has a crystal ball. It always feels like other people do though.
It's good to make decisions with a sober view of risk and reward. I feel so much market advice that filters down to us laypeople is whether we should or should not buy or sell, when really it is all based on our level of risk tolerance, how much in terms of assets we have, what our time horizons are, what our expertise is. I would say a healthy young person with plenty of cash to burn might well look at buying stocks now. But a person nearing retirement, perhaps with health issues, they should probably board up and get ready for the storm. Most people below retirement age should probably just stop freaking out and keep contributing to their retirement plan as they (hopefully) have been doing, and make sure to have some cash on hand.
Our society does not do well with self-control or with gray areas. We want easy answers, which, unfortunately, don't usually exist.
I could be wrong, but I'd see survivor bias as "at the end of the day, everyone still in the game reports their results", not acknowledging that those doing really poorly might walk away from the game.
You are making a fundamental mistake with your assumption that there is a clear distinction between laypeople and experts in the stock-market, there isn't. Money managers, mutual fund managers and hedge funds managers make terrible mistakes and lose money all the time, and are usually the 1st to panic. So, there is no such thing as an "expert" in the stock market.
This is where the panic goes to though: they might have 2-3 months cash right now, but might feel like they need 12-24 months cash to ride things out. People start worrying they'll lose their jobs and then their runway is at the mercy of the market. If a significant chunk of your wealth is in your house, that isn't easy to get out of during a downturn. Your tolerance for risk goes even further down if you have a family. Even further if your spouse doesn't work.
These are rational decisions that look like panic selling.
Tech work is a field that likely won't have to worry about this though.
>Tech work is a field that likely won't have to worry about this though.
With tech stocks tanking and economic growth likely sluggish for at least the next eighteen months, I can see large tech firms putting a hiring freeze in place, making it more difficult to switch jobs and more difficult to get hired if you DO get laid off.
Moreover, all that RSU comp that makes such a big portion of many tech workers take home pay just took a 20% haircut from a few weeks ago. It could well go lower.
Then there is the startup scene. I can definitely see current events massively impacting funding (see the recent Sequoia post). This could quite reasonably cause many startups to fail just like they did in 2008.
The IPO market is also going to be hit. I don't see any major unicorns doing an offering until the recovery happens. A lack of IPOs means, again, that the people willing to contribute to earlier stages of the pipeline, dry up.
Then there are secondary and tertiary effects. If the average tech worker sees their total comp drop by 10% to 20%, and quite a few lose their jobs, do you think the home you just bought in SF is going to be worth more in a year, or will there be some reduction (in appreciation if not absolute values)?
Indeed, if anyone things the Virus panic is as bad as it's going to get they're going to get some pretty rude shocks over the next few months.
Right now it really isn't that bad. I'm in London and going to work normally, the city is busy and the trains are still full in the mornings. Our company is starting to migrate people to working from home though. So a proportion of each team is wfh and staying there, for 14 days or until otherwise notified, and more will probably follow not because there is immediate danger, but because we want to be well prepared when there is.
If the markets are like this when there are a few hundred cases in the UK and US, what will they be like when there are tens, or hundreds of thousands of cases and major cities are in lockdown?
Back to investing, stocks are down significantly and it's quite possible now is a good time to start buying, for those with the capital. Maybe spreading out a balanced equity investment portfolio over the next several months. I don't know if or when the market might go lower, or by how much, but it seems unlikely it's going to rally all that much given that we can be pretty sure there's plenty more bad news still to come. So investing too much now might miss further falls, but IMHO buying over several months is unlikely to miss the dip. That's how I see it anyway.
> If the markets are like this when there are a few hundred cases in the UK and US, what will they be like when there are tens, or hundreds of thousands of cases and major cities are in lockdown?
Investors are trying to judge the likelihood of these very events and selling / buying accordingly. Markets are based on predictions.
Exactly. When the virus is at its worst is when the market will start to bounce back because recovery will be priced in. The markets are all about the value of the future. That’s why it’s all kind of a joke to me. Its based on hopes and dreams(or fears and nightmares) instead of how a company is actually doing.
Of course, but those predictions are only based on available information. We probably won’t know which companies will be worst affected, or where. There’s no way anyone could have predicted Lombardy would be the first part of Europe to be hit so hard. When particular companies, regions and industries turn out to be more severely affected, they will take the hit asymmetrically, but that will hurt the overall market. The current impact us being gauged fairly tentatively IMHO.
For example it’s likely several more airlines will go to the wall, but we don’t know which ones yet.
> Right now it really isn't that bad. I'm in London and going to work normally, the city is busy and the trains are still full in the mornings.
I flew from Seattle to San Francisco and back this weekend. In both cases, my "prime time" flights were less than 1/3 full. Returning yesterday, there was literally zero wait at bag check and security, I walked straight up to an agent. Haven't seen an airport like that in years, let alone a major.
> If the markets are like this when there are a few hundred cases in the UK and US, what will they be like when there are tens, or hundreds of thousands of cases and major cities are in lockdown?
Some conservative predictions are arriving at millions of cases in a couple of months. There are too many unknown variables for an accurate prediction but, without really drastic measures, it's difficult to argue with exponential growth.
At some point it will plateau and become another seasonal flu. And we may even get vaccines. But I agree that things are going to become way worse before they improve. Most worrying is the strain on the health care system, which may by itself worse prognosis on other, otherwise unrelated, diseases.
I agree I bought some of the early dips to later discover that it wasn't even closer to the bottom. So I chilled out and just paused on any further investment until the situation gets clear. I think US specifically has not seen yet the spread of the coronavirus, but it will come and things will get worse.
Maybe it is irrational buyers who are getting the benefit this time. A lot of unknowns right now. Past performance does not always predict future gains.
I remember the drugs company Elan[1] here in Ireland. It got hit bad in the early noughties by accounting issues and other problems, but they had this potentially killer drug called Tysabri and so, as they tanked, there were "smart" investors at every drop buying in and it just kept dropping and kept dropping. Loads of fairly smart people lost a lot of money. Only those who could really follow it all the way to the bottom were ok.
The high was $74 per share. The low was about $2 a share.
On the East coast too, it's not that common. I once heard it by an overly pedantic gentleman... But his pronunciation made me giggle so I started using it.
That's not insight, that's malpractice. I mean, maybe the customers are wrong. But the broker's desire not to execute sell orders and feed a panic is absolutely self-interest, and if they're wrong (and they often are! We had a similar panic just last week that turned out not to have been "the bottom"!) then they've deliberately hurt their customers to try to protect their position in the broader industry.
And that's bad. And it's the kind of behavior we tend to see at these overheated moments where the market has gotten away from the fundamentals and all the players are trying to find "tricks" to keep the gravy train rolling.
I don't know a single broker or sales-trader who would intentionally reject orders from a well-meaning customer. Brokerages make money on commissions and order flow when customers trade. The author of this Bloomberg opinion article has a kind of wisenheimer sense of humor and is playing fast and loose with his comments, in an attempt to make a point about the nature of panic.
Failure in moments of market stress is the worst way for a brokerage to protect its position in the broader industry, because downtime damages customer confidence (long-term income) in addition to reducing order flow (short-term income).
Order flow is more valuable in moments of extreme volatility because spreads widen when the market goes nuts.
>I don't know a single broker or sales-trader who would intentionally reject orders from a well-meaning customer.
It happened aplenty in 2008, when banks were rushing to offload toxic crap that they owned on to the market FIRST, and only THEN were they willing to help their customers do the same.
You should not assume that you can reasonably control your assets in moments of volatility. You may not be able to control them at all.
Market access is a major component of liquidity. If your broker already holds a phone in either hand, he won't be able to pick up your line when the light next to your name flashes.
Imagine that you have all of your liquid net worth in GLD and a huge cholera outbreak hits lower Manhattan. Bad shellfish. GLD rallies and you want to close your position and take profits. The problem is, the largest firms that make markets electronically on GLD have sent their employees home and shut down for the week. Their absence from the market makes it difficult for you to get a fair price for your large position.
Imagine that you use Verizon for your mobile phone and internet service. You think their service is great, so you buy shares of VZ. One day, Verizon announces that due to a malicious hacker, their network is experiencing cascading nationwide hardware failures that will not be repaired for at least a week. VZ stock plunges, but you can't modify your position because you can't get on the internet or make calls.
A less fanciful example would be owning stock when trading has been halted by the exchange. Perhaps there exists a buyer during the halt who would be willing to trade at a price that is favorable to you, but that transaction cannot happen.
Tail risk, in some cases, is a scenario where stocks get cheap. In other cases, stocks can't trade at all. (And in the worst of situations, tail risk is a scenario where guns get expensive.)
I'd consider all of those example cases where it's not necessarily reasonable to control your assets. I will say the Verizon case is a bit fanciful because it should be pretty easy to hop on another network quickly for most people to bypass that small hurdle...
When we're talking about an enterprise scale electronic trading system that should already have tested these sort of load scenarios, especially in a world where AWS/Azure/etc. exist and elastic computing is the target roll out for applications that probably don't even need scaling, that's not a reasonable case. That's a case where something that should already have been planned for and tested thoroughly slipped by. It's especially damaging when it happens repeatedly over a short time span. There's nothing unpredictable here. Management and IT at Robinhood surely read the news and knew there would be massive load today.
Surely, someone designing systems and software in this domain understand that the market could be highly volatile and their system needs to be ready for those cases where it's flooded by customer requests or clearly conveyed to customers that it's not ready for such cases.
Robinhood operate on razor-thin margins and if you just look at where three-month implied vols were trading in early Jan, we are in a 7-sigma event. To be fair to Robinhood, it is probably hard to anticipate and prepare for such an event when you're not charging commissions. They are not the only brokerage that has had problems in the past seven days. And they lose money when their customers can't place orders.
"You should assume you can reasonably control your assets at any given time." Sometimes you can't reasonably control your assets, because you can't control them at all. So you shouldn't assume that you can reasonably control them in those times.
I hope it is clear that I am not trying to be argumentative; I actually think it's a complete let-down that Robinhood has left so many people frustrated as their wealth whip-saws around in the market. I'm just saying that tail scenarios don't usually fit into our predictive model of what's "reasonable," even when the solutions seem crystal clear in retrospect.
On a more general point, I never understood why market returns are assumed to be gaussian - these "one in a billion" events seem to happen far too often.
Business Adventures has a similar anecdote, referring to the 1962 crash:
> "Other firms were less fortunate, and in a number of them confusion gained the upper hand so thoroughly that some brokers, tired of trying in vain to get the latest quotations on stocks or to reach their partners on the Exchange floor, are said to have simply thrown up their hands and gone out for a drink. Such unprofessional behavior may have saved their customers a great deal of money."
"Oh, hey, we saw your message but we were just too overwhelmed to execute it. We did however, manage to do the next best thing, which was to purchase an option, expiring today, to sell at the prevailing price at the time of your call, and, to make things right, we'll go ahead and exercise that option, unless you'd prefer we just debit you the cost of the option and let it expire without exercise?"
Honestly, I think this is BS that a broker/platform can just cut off investors from making buy/sell decisions when it’s inconvenient (unprofitable) to handle.
Brokerages aren’t doing investors any favors that are purposefully trying to buy a dip or sell for cash, they are only doing themselves favors as the middle man.
Lame arguments from brokerages don’t change that. Has there been a recent case of “unexpected” downtime during a huge market rally?
It's pretty easy to find yourself creating bottlenecked points of failure in systems that expect consistency. There may not be the knobs necessary to scale up past a certain point.
A sufficiently bad positive feedback loop can also drive you into the ditch no matter how monstrous your hardware is.
It's always been either completely down or having a huge lag in executing orders at market open for the last 6 business days. That leap year bug was a hoax. Robinhood just can't scale under heavy load.
I wonder if people will leave the platform en masse to alternatives as most brokers have no fees on trades these days.
I don't see how it's worse to do it on a phone than your laptop, or over the phone with a broker via a voice call?
Most folks I know who use RH just use it as a fun little side gamble/game. Small-ish amounts of money, something they could afford to lose, with the rest in more stable funds or investments.
I agree, both that that's my experience and that it's the right way to use RH. But then that means this isn't exactly an existential crisis; if you can't run your side gamble during some periods of high market volatility, does it really matter?
On one hand, now is a fun/important time to be messing with your side gamble. It'd be like if a betting site went down the day of the Super Bowl. If you're just using RH to gamble and day-trade, periods of high volatility are some of the most fun times to be actively trading!
On the other hand, I don't really care. I'll be fine. Most of the folks I know who use RH are just joking about it being down. It's annoying, but no one's broken up about it. A lot of the folks casting their anger at RH seem to be folks who already dislike it and probably don't use it?
Professional or large-scaled investors with large sums invested probably aren't using RH for a lot of reasons even before you get to the downtime issues.
The company never said their downtime was related to the leap year afaik, that was some random twitter thread looking at requests on their browser that had datestamps from the future (which could have other explanations). Unless they're rolling their own date library (which is very unlikely), the code should be using the system time of whatever computer the software is running on, meaning it would be extremely unlikely that the leap year caused a bug.
You are correct in that it's more likely their system just can't scale.
I did just that early last year when I realized that they simply don’t have the infrastructure, support personnel or sense of urgency I needed from a broker to handle my money. This is pre-free trades — I decided that I rather pay a fee than roll the dice on when they’ll clear trades and bank transfers (in one instance an ACH took 2-weeks, with canned messages from support).
In startup land, a junior developer is another startups senior developer, and same applies to senior developer <> VP. Simply down to what the startup you're working at. Might not be that far.
It's just a weird financial thing. Everyone gets to be a VP. I was actually a bit embarrassed by having the title when I worked at BofA. It's not even a manager level. I was what would be a senior developer anywhere else.
The simple way to get a job like that, is to apply for it. Instead of waiting for someone to retire in your current company for instance.
And its not unheard of. My sister-in-law worked from startup contributor to Cisco VP by applying for the next job up, one at a time, from company to company.
In the big-name Wall Street banks 1/3 of employees are VP or higher.
Senior Developer in the "outside world" typically means Associate in the banks, and VP is the next rung up from Associate, so VP really is just one hop from Senior Developer.
Honestly, I don't. There is enough public information about the brash attitudes of this company's execs that anyone working for a company like this should know that the lowest level engineers working on a technical problem like this are eventually just going to get thrown under the bus.
I hope this is a lesson learned for smart and impressionable engineers about where to work.
The implication from the parent comment is that these individuals are going to go through some emotional strain.
My point - it's difficult for me to feel the same anxiety someone might feel in this situation when they equally ignored the risks of something that is easily identifiable. They very likely felt elated when they joined because the company was seen as cool, hip, and by all objective means is/was successful (valuation, # of users, etc).
It is not, was just confirmed on CNBC that Robinhood is experiencing system wide issues long after the market reopened...
Frankly, if I am an user of this platform, I'll get my money out first thing after the fix the access...
If the can't keep the system running straight for so many days, who knows what other bugs they have over there...
This is a good reminder that all these new mobile(brokers|banks|&c.) are not as good as the historical systems that have been stress tested for decades. All is good when everything works as expected but as soon as the slightest issue arise you're in for a ride.
My direct trading account at my (very) old bank is also saying it's having technical difficulties today. It sounds like display issues, not trade execution issues though.
I know Fidelity changed their default dashboards to not load data-heavy views for the past week or two. Usually you get a nice dashboard page with some charts and personalized info, but right now you get a few generic (non-customized) market stats and a bunch of generic links.
Still useful enough, and if you want, you can get to your own dashboard... but things like this (inconsequential as they seem) can be the difference between going down or not.
Each non-personalized element on the page that's removed saves at least one DB query (and in a fancy app like Robinhood's case, maybe a half dozen).
That's what I think of every time I see a post about some new VC backed startup that says they are going to disrupt an industry by creating a lean product that cuts out all of the unnecessary cruft and legacy systems. People need to learn that there's usually a reason something is done a certain way, and having barrels of VC money doesn't make you an expert in a field.
Pasting this has recently become a meme. Not sure it applies here, as it's not about removing stuff from the same instance, but about a competitive lean product?
It's about removing legacy systems and pretty directly about reforming xyz industry with the new tech, so I think the principle of reform requiring knowing why something you want to get rid be of is there in the first place is directly related to the topic
A quirk of information propagation. If it's there, it was put there for a reason. If you don't understand the reason, you're at risk of having to rediscover it the hard way; something we generally wish to avoid at all costs.
Which is somewhat ironic considering the attitude around things like Trade Secrets leading to a plethora of Chesterton's Fences being basically guaranteed to litter the economic landscape.
I think the real irony is that much of the time fences exist because of the quirks (fences) of some other system you are stuck with. Sometimes it feels like it is fences all the way down. Other times, you work your way down 5 layers of dependencies and discover that the whole pile of ugliness is because the core system is designed around physically mailing boxes full of reel-to-reel tapes. When that is the case, there is potential for a cascade of "lean modern apps" to de-cruft an industry.
Ultimately, RH is backed by a legacy system somewhere. I believe they custom built their OMS (order management system) which was the previous bottleneck but somewhere between RH and the other side of the order is a legacy system that they can't disrupt.
Sometimes the reason it's done that way, is because they were in a rush in the first iteration, and never got around to refactoring.
Loads of VC money means you can take more time up front to architect things better. Maybe have more experiences engineers and architects on the team from the start.
Not really. VCs expect 10x returns. Yesterday. They really don't care if it's the best system ever designed. It just has to work until a liquidity event.
I understand your point, but I think VC backed companies are likely in a more hyper growth mindset than banks and conservative industries were when they were making these systems in the 80’s.
As much flak as I’ll get for saying this, why Python? It seems like Robinhood went all in on an ecosystem that is easy to get an MVP off of the ground with, but certainly not as fast and reliable as you would expect a system moving billions everyday. Language typically doesn’t matter for end users, but man, there are at least 10 solid options to build a critical trading system in that are faster and safer than Python.
Is it really 100% python? I can't find any indication of what they use besides the python helper library for Kafka. Seems like a message queue would be conducive to partial rewriting of the system.
This might be slightly unfair to Robinhood ... the time we saw volatility similar to this was in 2008. Most of the major banks had technical difficulties that day (despite being around for decades of years).
Technology and scaling are tough. Reddit, Twitter and CloudFlare have gone down in the last 12 months, despite being constantly stress-tested.
Why not? I think now would be the most interesting time to be working there, where you can learn the most in the smallest amount of time. You'd gain much more knowledge powering through the hard times rather than just coasting around for the easy ones.
Yeah but what you’re learning is “how to not catastrophically fail your customers”. In a better company that’s a lesson you already learned. There is going to be a LOT of stress and tension, it's hardly a great learning environment.
>In a better company that’s a lesson the company already learned.
FTFY. Just because you work at a company doesn't mean you were around to experience what ever event provided some sort of "learning" experience. If you don't experience it personally, what ever systems the company has in place to protect itself from said events just look like bloat/legacy cruft ripe for a young upstart to disrupt away.
If RH had any interest in being a sustainable business that respects their customers they would have hired engineers that have already been through the "hard times", several times, and don't need to learn on the job. But in startup land sustainable businesses are not the goal - customers be damned.
And this is what happens when you live wildly on the brink of tech fantasy and tech brilliance.
I believe this will be Robinhood's undoing and the company will likely just firesell itself (less than $1B) to Schwab/TD so they can acquire the younger demographic.
Does anyone have a good path forward of how Robinhood can even survive after this? Other brokers are not going down, and also offer free trades. I don't even see the point of it anymore, especially if it is going down in some of the most important market moments.
I have a crazy strategy, but hear me out: They do nothing.
The vast majority of Robinhood's customers aren't day trading clamoring to get their daily dose of tendies. The proportion of RH's customers who are actually effected is probably quite minimal, and of those some might change to another broker, but most will probably just put up with it. The market isn't going to be this active for long, and so once it calms down RH will put in place some fixes for its systems. This will be forgotten after a couple of weeks, it'll perform better next time and it'll have almost no impact on RH's value.
Oh, is it 10:45 am on a Monday already? After the first two days, I downloaded another broker and determined to transfer, but it's kind of a pain. Guess I have to actually do it now.
This many incidents makes FINRA a joke. Time for some outside regulators to step in and require certain levels of availability with real consequences if not maintained.
Merrill Lynch / Bank of America's trading platform is also down... so this isn't limited to Robin Hood. I was completely unable to place any trades or even check balances during the market opening this morning.
However, engineering HEADS SHOULD ROLL. In the era of on demand computing, near limitless cloud resources, etc, these engineering teams should have already had capacity planning ready to go for these trading platforms. I'm extremely disappointed - these engineers LIVE for this stuff, and they dropped the ball.
Again, heads should roll. This is not some data center that caught fire, this is capacity planning at its lowest common denominator.
The market hit a circuit breaker this morning when it was 7% down. No one could trade at the open, not even professionals, because the market was stopped.
No, I couldn't event place any orders whatsoever, and couldn't check balances etc. Order PLACEMENT and order execution are two different things. The former was impossible as well.
Agree that heads should role, but it may very well be management decisions that brought them to this point. We don't know if engineering was resource/strategically limited (seems unlikely at a well funded startup but it may have been). Having worked places where terminations were the result of failures, it's not always the party most responsible that is let go.
Sometimes engineers are not at fault. They are well aware of technical debt and possible performance mine fields. It's the management that does not properly prioritize performance and scale work because there is no apparent "value" to the customer. Until it's too late.
Bad idea. You would be just rolling heads of people who just learnt a valuable lesson in high throughput services. Keep them around, they'll get better than hiring someone else who will have to spend more time learning the same lesson.
Designing a real time system to scale to this magnitude is not easy at all. I wonder how many engineers would be capable of designing such a system, I sure couldn't. This isn't like most scaling where delaying/queuing can be an acceptable trade off.
Maybe they shouldn’t be in business if they can’t hire the engineering to do the job. Every other discount broker seems to have no problem hiring this level of talent.
1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely.
2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info?
I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.
he is the customer, not a employee. As a customer I don't actually care about postmortems, i just want things to work.
If it doesn't work, I use another product. I'm leaving robinhood due to their issues, and it sucks it takes so long due to their transfer limits
That's fine and for every customer to evaluate for themselves. But there is a difference between saying "as a customer the product isn't acceptable so I will use something else" and "engineering is incompetent and someone should be fired". Especially if you don't actually know what happened.
There should be accountability for sure. I am not sure if rolling heads is enough or even the right thing by itself - but there definitely needs to be a 3rd party audit, and a public report.
FWIW, Charles Schwab is also down, and is often down during heavy trading. There's a big reliability gap between straight consumer brokers, and the next tier up with interactive brokers et al.
yes, this. many other systems go down in explosive volatility, including IB, tda thinkorswim, etc and they all have their ass covered with electronic trading risk agreements but you dont hear anything about that on wsb because theyre all casual RH chunkers
I just wrote a post about /r/Wallstreetbets reaction to this. I graphed out comments for several different brokers over time, and also mapped how many users were holding SPY puts and got trapped.
346 comments
[ 2.4 ms ] story [ 326 ms ] threadSee replies to Robinhood's tweet: https://twitter.com/AskRobinhood/status/1237016846282280961
https://techcrunch.com/2020/03/03/robinhood-outage-cause/
If this was Wells Fargo we'd see the pitchforks. But it's a SV company so all is good.
Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.
IMO anyone who opened a position on RH in the last week should not be shocked that the same thing happened again in this high volume crapstorm.
Provision the resources and take the financial hit in costs.
If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent.
Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so.
> Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.
ftfy
Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.
$0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.
Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money.
Comparing it with a site like reddit is, frankly, a bit disengineous.
Your customers are paying for speed and performance .
In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market
The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.
Nearly every write-action on Reddit appears to hit a queue. If your vote or comment doesn't go up for 30 seconds, that's not a problem.
If your trade takes 30 seconds, that could be a huge deal.
Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.
Please tell me you do not manage a team of software developers.
Rolling out fixes fast, even if they require intensive changes, is completely reasonable and expected in many industries though. They should have the talent and procedures to get it done. This isn't some startup web app, it's a multi-billion dollar broker managing millions in client funds. It's bordering on incompetent to have 3 outages in 2 weeks.
EDIT: What exactly is everyone disagreeing with?
This is a financial trading platform. Do you understand the risks of potentially introducing a different bug?
Changing a single line can introduce a different bug. Use proper QA and testing to catch as many as possible as with any development.
My emphasis is on getting things fixed quickly. They need to do whatever it takes to get systems online asap. Not sure what's so controversial about that.
I'm surprised by all the misinterpretation in this thread. Seems like it reflects the laid-back West Coast/SV attitude that isn't a good fit for high pressure time-sensitive work in other industries.
This is the part you don't understand. There is a difference between digging ten one-foot deep holes vs one ten-foot deep hole. People need time to plan how to coordinate and then get on the same page so that everyone can work at their own pace. That is the part that is not parallelizable and is the rate-determining step.
If this sounds unfamiliar or onerous then it's because you and others might have never experienced teams that do this. Robinhood is clearly lacking this experience and disaster planning.
RH wasn't prepared with any contingency. They should have a resolution for their users - even if they can't find or fix the original cause. That's the failure I'm talking about.
See the 2 other users in this thread that describe similar high-pressure situations.
Reality is, without knowing more about what’s causing this it’s impossible for either of us to say. If there is indeed some fundamental bottleneck that was previously not known, then I certainly won’t be surprised if it takes a while to sort out.
Now you can say they should’ve load tested, capacity planned etc etc. But we are where we are. Still can’t go back in time to turn this into a quickly fixable problem if it’s currently not.
Edit: also pretty disappointed that we don’t know more about the root cause. As an user I’d want to know what the issue was and what they are planning to do to about it to evaluate if I should trust them going forward.
So I'm not sure if I know that the state of the art for trading platforms is as rock solid as everyone is implying, and Robinhood seems to be way far off from whatever gold standards there are, see infinite leverage bug. So I don't think it's crazy that they move quickly to fix it.
I'd never be able to stomach the pressure, and I wouldn't wish it on others, but it doesn't seem crazy.
There are many ways to resolve these issues FAST. The easy way is to throw money at this. Go tomorrow (literally) and bypass all procurement controls and go to a mega big provider and scale this asap. Any financial services with a half decent IT has done the paper exercise to this scenario (at least for the purposes of BCP/DRP). The slow/better/mature way may be too slow, especially with the current market conditions.
The RH folks will definitely get a visit from SEC, their external auditor, and their external auditor will get a visit from SEC.(their auditor will be in the deepest of shits)(how come they failed to spot such a going concern issue?)(what the hell were they looking for on their audits?)(did they only send juniors over there?)
I feel sorry for the retail traders that got knocked down. I think that anyone locked in buying at 27-28k (US30) should wait 6 months to breakeven and after the US elections (irrespective of the winner) there will probably be a rally.
They are probably at ~10X peak traffic at times.
Just are not prepped for a hardcore crash.
[1] https://www.ccn.com/dow-crashes-so-hard-might-close-stock-ma...
[2] https://www.businessinsider.de/international/stock-market-ne...
[3] https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u...
Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.
It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.
What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.
> It is well known that one of the best services a retail broker can provide is not answering the phones during a crash. The market is down, the customers panic, their timing is terrible, they want to sell at the bottom, they call you up to say “sell everything,” you say “we’re sorry all our representatives are assisting other customers, your call is important to us,” they hang up and get distracted, the market rallies, they forget about selling, you have saved them a fortune, good work.
[0] https://www.bloomberg.com/opinion/articles/2020-03-03/robinh...
In the movie (based on real events) the same thing happened: during the housing crash suddenly every bank had 'technical issues' when people tried to sell their stocks.
[0] https://en.wikipedia.org/wiki/The_Big_Short_(film)
Disclaimer: I'm not in finance, don't know much about it, but it's what I understood while watching the movie ;)
Yeah, right, it's so nice platforms prevent people from selling, just as the big fish unload their own stocks.
Reality is the regulations on the financial system are a feature. Being able to have charges reversed if I get my card skimmed and someone makes a bunch of charges is a feature. Being able to get my money back if the seller ships me broken goods (or nothing at all) is a feature. Having regulations on banks front-running their customer's transactions is a feature (one that Robinhood unfortunately gets around somehow). Having regulation on securities not being scams is a feature.
As with anything, the answer is sometimes.
You want those rules for your retirement account, because if someone can steal a six figure sum from you instantly with no takebacks, that is bad.
But the overhead of that system is not always required, and it's the major reason that we don't have e.g. micropayments. The overhead of allowing transactions to be reversed eats the entire transaction amount for very small transactions.
So it would be good to have a system where you could, for example, transfer $50 into it from your bank account, wait the month or so until the transaction can no longer be reversed, and then transfer it from there a few pennies at a time with insignificant transaction costs because the small transactions are instantaneous, anonymous and can't be reversed. Which isn't nearly as problematic because even if you get scammed, your loss is limited to the $50 you put in. Instead of getting wiped out, you pay a reasonable cost for a personal lesson on security and trust.
And there is no reason we couldn't have both. Then you keep the bulk of your wealth in the inefficient safe system and some petty cash in the one with lower transaction costs and get the best of both worlds. But the existing rules don't allow that, which is bad.
For example, if we took a simpler example of something more people are familiar with like options...let's say you buy a put at a strike of 100. The stock is trading right around 100. The market tanks like it has been doing and the stock goes down to 80. That's great for you since you had the put. You technically have the right to buy the shares at the market and shove them onto someone and force them to pay you 100. Most of the time, since we're dealing with derivatives, they just trade the value of the option itself and skip dealing with the underlying security because it takes a lot of capital (100 shares * 80 bucks each in this case). So the value of the option (the derivative) should be around $20 as you approach expiration and the premium fades off. If the market maker for the options has way too much exposure and wrote naked puts (meaning they didn't have the underlying security) they took on a ton of risk and basically got fucked. In this scenario it's like Burry calling and finding out his options are worth like $1.50 instead of $20 simply because they are thinly traded and therefore can't get a good price by anyone. This actually happens on some options - you'll see a huge spread between the bid and ask on some options so it's not like it doesn't happen even for this type of retail-friendly derivative.
In the Big Short case, the banks were acting as market makers, or more technically in this case dealers- this was more akin to going to a used car lot. You read a lot of reports saying that the new Fords were flying off the lots for outrageous prices, and to people with dubious ability to pay for the sticker prices they did because they got loans they shouldn't have. You go to the dealer and say I want to sell these cars short and buy them back later. The dealer presumably thinks you are an idiot, or wants to hedge some risk (the analogy is breaking down here), but says ok sure, thinking he is going to sell them for even more in 6 months. 6 months later comes- you were right- there is a flood of Fords on the market at half the price because so many people had them repossessed or are desperately trying to get out of these loans that are killing them. You go to the dealer and say "Ok bud, I'd like to buy these cars back at half price..." and they say "Nah, these are still worth 98% of what you sold for... how about that price?" and you are kind of stuck. You bought specific used cars that are kind of but not really entirely fungible. You can't just go down the street and buy those cars and replace them. You have to hope that they feel the pressure to lower those prices because they start feeling the squeeze for cash, or a regulatory agency comes in and puts the pressure on. They can kind of live in la-la land and avoid the reality of the situation as long as they have no requirement to sell.
Eventually though, they blinked and once one bank started taking write downs, all banks did, and once that became acceptable, they all followed suit- and they also needed the cash at that point.
Anyway- now back to stocks/options- these have well known discoverable prices and you can sell them on open markets where the brokers you connect to might be on the other end of the deal but its really unlikely. A broker/dealer like Robinhood or Schwab or Interactive Brokers should theoretically have an entirely flat position at the end of the day.
TL,DR: Robinhood didn't go down because it doesn't want you to sell your stocks.
This means that transaction fees spike to 10-100x their normal rate during those crises, or else your transaction just sits and takes a week to get confirmed (if it ever does at all - eventually it will just time out after a few weeks).
As much as I love Rails, it's extremely hard to scale an application written in it to handle millions of users. All the Active Record transactions have to be rewritten as database transactions if you want to make sure that the databas doesn't lock the tables for a long time.
Groupon wasn’t anywhere near real-time that a trading system requires
The choice of a language or framework is only one factor on the success of failure of a system.
You can write a broker in brainfuck if you are so inclined. You can make it rock solid. Your pick may require more or less engineering resources, but that's all it does.
You mention ActiveRecord. That too is premature optimization. Crafting optimized SQL queries will not save a badly architected app. They might even be worse if whoever is optimizing them doesn't fully understand what they are doing.
> if you want to make sure that the databas doesn't lock the tables for a long time.
You didn't mention which database you are using. You didn't say anything about its schema. This is yet another generalization. Not all databases will "lock the tables", no matter how badly queries are written. And, if they do, optimize that.
Usually, only a handful of services are critical. Microservices are all the rage, optimize those to death.
When volumes go up, the servers run out of money, and everything grinds to a halt till humans intervene.
The humans typically want to do a lot of checks they aren't being tricked into refilling the online wallet too. Those checks are usually manual and take time.
Not being able to exit these high-risk positions won't just bring your account to zero, but potentially put you in the six-figure negative if you are unlucky.
I want to be on the other side of a LOT of trades today and I'm unable.
The future is unknown to all of us. In some cases, looks are not deceiving. When the market appears to be on the cusp of freefall, it can in fact accelerate downward.
(1) They trade on margin, so huge drops are much more dangerous to them
(2) Their boss sees a Profit & Loss for the trading day that evening, if they don't like what they see they might give their portfolio to somebody else.
(3) Customers are calling the sales desk and they want to sell; hopefully sales can slow them down, but you may need to sell to pay for redemptions. Many customers may need to sell to rebalance their portfolios, get liquidity, etc. This is a mechanism which can carry instability from one market into another one which would otherwise be doing just fine.
There are a handful of hyper-geniuses that can make gobs of money by actively trading. For the rest of us, there's buy and forget.
Because the Harvard endowment, like most all other funds out there, generally fails to beat the market and has had down years even while the market was up. This is probably due to lower risk tolerance, but it doesn't really show that hyper geniuses are running it either.
Where I live real estate was massively overpriced, so still hasn't rebounded, more than 10 years later.
* this is obviously the caveat and where things went wrong for people.
You're also forgetting that they had somewhere to live for a decade for the cost of inflation of the house, which is significantly cheaper than normal rental prices in many places.
To come out even after a decade, after housing costs, seems a pretty "safe" investment to me. Certainly not as lucrative as having invested in an index tracker, but it beats the 0% interest my bank pays on my current account balance.
Why?
I am not an economist though, this is just my layman’s view.
What you actually have to figure out is: is there any unexpected future efficiency gains that the market doesn't expect. You have to know something that the market does not.
Isn't this ignoring opportunity cost, uncertainty, etc.? If the expectation of future gains is priced in, wouldn't treasury bonds sell for face value + remaining interest on the secondary market?
You don't actually need to improve efficiency/productivity if the government has tools like the discount rate, open market operations, the ability to alter margin requirements on derivatives, fed funds rate, as well as programs and policies like MMIF, TAF, CPPF, ABCP, TALF, ZIRP, to manipulate the money supply and the velocity/flow rates of money.
Assuming you have no pressing needs for your invested capital (e.g. most retail investors), the market cannot actually stay irrational longer than you can remain solvent.
Well, the companies that don't go bust.
It's good to make decisions with a sober view of risk and reward. I feel so much market advice that filters down to us laypeople is whether we should or should not buy or sell, when really it is all based on our level of risk tolerance, how much in terms of assets we have, what our time horizons are, what our expertise is. I would say a healthy young person with plenty of cash to burn might well look at buying stocks now. But a person nearing retirement, perhaps with health issues, they should probably board up and get ready for the storm. Most people below retirement age should probably just stop freaking out and keep contributing to their retirement plan as they (hopefully) have been doing, and make sure to have some cash on hand.
Our society does not do well with self-control or with gray areas. We want easy answers, which, unfortunately, don't usually exist.
That's because you hear them when they are making gains, never when they are losing.
I could be wrong, but I'd see survivor bias as "at the end of the day, everyone still in the game reports their results", not acknowledging that those doing really poorly might walk away from the game.
This is where the panic goes to though: they might have 2-3 months cash right now, but might feel like they need 12-24 months cash to ride things out. People start worrying they'll lose their jobs and then their runway is at the mercy of the market. If a significant chunk of your wealth is in your house, that isn't easy to get out of during a downturn. Your tolerance for risk goes even further down if you have a family. Even further if your spouse doesn't work.
These are rational decisions that look like panic selling.
Tech work is a field that likely won't have to worry about this though.
With tech stocks tanking and economic growth likely sluggish for at least the next eighteen months, I can see large tech firms putting a hiring freeze in place, making it more difficult to switch jobs and more difficult to get hired if you DO get laid off.
Moreover, all that RSU comp that makes such a big portion of many tech workers take home pay just took a 20% haircut from a few weeks ago. It could well go lower.
Then there is the startup scene. I can definitely see current events massively impacting funding (see the recent Sequoia post). This could quite reasonably cause many startups to fail just like they did in 2008.
The IPO market is also going to be hit. I don't see any major unicorns doing an offering until the recovery happens. A lack of IPOs means, again, that the people willing to contribute to earlier stages of the pipeline, dry up.
Then there are secondary and tertiary effects. If the average tech worker sees their total comp drop by 10% to 20%, and quite a few lose their jobs, do you think the home you just bought in SF is going to be worth more in a year, or will there be some reduction (in appreciation if not absolute values)?
Right now it really isn't that bad. I'm in London and going to work normally, the city is busy and the trains are still full in the mornings. Our company is starting to migrate people to working from home though. So a proportion of each team is wfh and staying there, for 14 days or until otherwise notified, and more will probably follow not because there is immediate danger, but because we want to be well prepared when there is.
If the markets are like this when there are a few hundred cases in the UK and US, what will they be like when there are tens, or hundreds of thousands of cases and major cities are in lockdown?
Back to investing, stocks are down significantly and it's quite possible now is a good time to start buying, for those with the capital. Maybe spreading out a balanced equity investment portfolio over the next several months. I don't know if or when the market might go lower, or by how much, but it seems unlikely it's going to rally all that much given that we can be pretty sure there's plenty more bad news still to come. So investing too much now might miss further falls, but IMHO buying over several months is unlikely to miss the dip. That's how I see it anyway.
Investors are trying to judge the likelihood of these very events and selling / buying accordingly. Markets are based on predictions.
For example it’s likely several more airlines will go to the wall, but we don’t know which ones yet.
I flew from Seattle to San Francisco and back this weekend. In both cases, my "prime time" flights were less than 1/3 full. Returning yesterday, there was literally zero wait at bag check and security, I walked straight up to an agent. Haven't seen an airport like that in years, let alone a major.
Some conservative predictions are arriving at millions of cases in a couple of months. There are too many unknown variables for an accurate prediction but, without really drastic measures, it's difficult to argue with exponential growth.
At some point it will plateau and become another seasonal flu. And we may even get vaccines. But I agree that things are going to become way worse before they improve. Most worrying is the strain on the health care system, which may by itself worse prognosis on other, otherwise unrelated, diseases.
Why not? Assuming you can weather the bear market, the bulls will eventually return, no?
The high was $74 per share. The low was about $2 a share.
[1] https://en.wikipedia.org/wiki/%C3%89lan
I’m on the East Coast of the US, and though it makes sense, I haven’t seen it before.
It was was also used for the 1900-1909 decade as well.
And that's bad. And it's the kind of behavior we tend to see at these overheated moments where the market has gotten away from the fundamentals and all the players are trying to find "tricks" to keep the gravy train rolling.
Failure in moments of market stress is the worst way for a brokerage to protect its position in the broader industry, because downtime damages customer confidence (long-term income) in addition to reducing order flow (short-term income).
Order flow is more valuable in moments of extreme volatility because spreads widen when the market goes nuts.
It happened aplenty in 2008, when banks were rushing to offload toxic crap that they owned on to the market FIRST, and only THEN were they willing to help their customers do the same.
Let people sell, that's on them.
Market access is a major component of liquidity. If your broker already holds a phone in either hand, he won't be able to pick up your line when the light next to your name flashes.
Imagine that you have all of your liquid net worth in GLD and a huge cholera outbreak hits lower Manhattan. Bad shellfish. GLD rallies and you want to close your position and take profits. The problem is, the largest firms that make markets electronically on GLD have sent their employees home and shut down for the week. Their absence from the market makes it difficult for you to get a fair price for your large position.
Imagine that you use Verizon for your mobile phone and internet service. You think their service is great, so you buy shares of VZ. One day, Verizon announces that due to a malicious hacker, their network is experiencing cascading nationwide hardware failures that will not be repaired for at least a week. VZ stock plunges, but you can't modify your position because you can't get on the internet or make calls.
A less fanciful example would be owning stock when trading has been halted by the exchange. Perhaps there exists a buyer during the halt who would be willing to trade at a price that is favorable to you, but that transaction cannot happen.
Tail risk, in some cases, is a scenario where stocks get cheap. In other cases, stocks can't trade at all. (And in the worst of situations, tail risk is a scenario where guns get expensive.)
When we're talking about an enterprise scale electronic trading system that should already have tested these sort of load scenarios, especially in a world where AWS/Azure/etc. exist and elastic computing is the target roll out for applications that probably don't even need scaling, that's not a reasonable case. That's a case where something that should already have been planned for and tested thoroughly slipped by. It's especially damaging when it happens repeatedly over a short time span. There's nothing unpredictable here. Management and IT at Robinhood surely read the news and knew there would be massive load today.
Surely, someone designing systems and software in this domain understand that the market could be highly volatile and their system needs to be ready for those cases where it's flooded by customer requests or clearly conveyed to customers that it's not ready for such cases.
"You should assume you can reasonably control your assets at any given time." Sometimes you can't reasonably control your assets, because you can't control them at all. So you shouldn't assume that you can reasonably control them in those times.
I hope it is clear that I am not trying to be argumentative; I actually think it's a complete let-down that Robinhood has left so many people frustrated as their wealth whip-saws around in the market. I'm just saying that tail scenarios don't usually fit into our predictive model of what's "reasonable," even when the solutions seem crystal clear in retrospect.
On a more general point, I never understood why market returns are assumed to be gaussian - these "one in a billion" events seem to happen far too often.
> "Other firms were less fortunate, and in a number of them confusion gained the upper hand so thoroughly that some brokers, tired of trying in vain to get the latest quotations on stocks or to reach their partners on the Exchange floor, are said to have simply thrown up their hands and gone out for a drink. Such unprofessional behavior may have saved their customers a great deal of money."
"Oh, hey, we saw your message but we were just too overwhelmed to execute it. We did however, manage to do the next best thing, which was to purchase an option, expiring today, to sell at the prevailing price at the time of your call, and, to make things right, we'll go ahead and exercise that option, unless you'd prefer we just debit you the cost of the option and let it expire without exercise?"
'Debit! Debit! Just debit!'
Brokerages aren’t doing investors any favors that are purposefully trying to buy a dip or sell for cash, they are only doing themselves favors as the middle man.
Lame arguments from brokerages don’t change that. Has there been a recent case of “unexpected” downtime during a huge market rally?
At least in such a volatile time period.
A sufficiently bad positive feedback loop can also drive you into the ditch no matter how monstrous your hardware is.
I wonder if people will leave the platform en masse to alternatives as most brokers have no fees on trades these days.
Most folks I know who use RH just use it as a fun little side gamble/game. Small-ish amounts of money, something they could afford to lose, with the rest in more stable funds or investments.
On the other hand, I don't really care. I'll be fine. Most of the folks I know who use RH are just joking about it being down. It's annoying, but no one's broken up about it. A lot of the folks casting their anger at RH seem to be folks who already dislike it and probably don't use it?
Professional or large-scaled investors with large sums invested probably aren't using RH for a lot of reasons even before you get to the downtime issues.
It probably works fine, but I am never as confident on a touch screen. I even prefer to order pizza on a desktop.
You are correct in that it's more likely their system just can't scale.
After vents like this juniors turn into seniors, seniors into VPs, and VPs going on sabbatical.
I was an intern in a team where everyone who's been in the company for more than a year was a VP.
And its not unheard of. My sister-in-law worked from startup contributor to Cisco VP by applying for the next job up, one at a time, from company to company.
Senior Developer in the "outside world" typically means Associate in the banks, and VP is the next rung up from Associate, so VP really is just one hop from Senior Developer.
Just how they do things there.
Honestly, I don't. There is enough public information about the brash attitudes of this company's execs that anyone working for a company like this should know that the lowest level engineers working on a technical problem like this are eventually just going to get thrown under the bus.
I hope this is a lesson learned for smart and impressionable engineers about where to work.
My point - it's difficult for me to feel the same anxiety someone might feel in this situation when they equally ignored the risks of something that is easily identifiable. They very likely felt elated when they joined because the company was seen as cool, hip, and by all objective means is/was successful (valuation, # of users, etc).
Still useful enough, and if you want, you can get to your own dashboard... but things like this (inconsequential as they seem) can be the difference between going down or not.
Each non-personalized element on the page that's removed saves at least one DB query (and in a fancy app like Robinhood's case, maybe a half dozen).
Innovators dilemma?
Which is somewhat ironic considering the attitude around things like Trade Secrets leading to a plethora of Chesterton's Fences being basically guaranteed to litter the economic landscape.
The problem is RH itself.
It's like saying Airbnb is taking down root level DNS servers.
I didn't realize they were a big player in this game. Citadel definitely is, though.
This is strictly an RH problem. Their last blog post said it was a DNS problem.
Since all securities are affected, it probably actually is their OMS which is malfunctioning.
Loads of VC money means you can take more time up front to architect things better. Maybe have more experiences engineers and architects on the team from the start.
As much flak as I’ll get for saying this, why Python? It seems like Robinhood went all in on an ecosystem that is easy to get an MVP off of the ground with, but certainly not as fast and reliable as you would expect a system moving billions everyday. Language typically doesn’t matter for end users, but man, there are at least 10 solid options to build a critical trading system in that are faster and safer than Python.
Technology and scaling are tough. Reddit, Twitter and CloudFlare have gone down in the last 12 months, despite being constantly stress-tested.
Especially not when this happens for the third time in a row.
FTFY. Just because you work at a company doesn't mean you were around to experience what ever event provided some sort of "learning" experience. If you don't experience it personally, what ever systems the company has in place to protect itself from said events just look like bloat/legacy cruft ripe for a young upstart to disrupt away.
I believe this will be Robinhood's undoing and the company will likely just firesell itself (less than $1B) to Schwab/TD so they can acquire the younger demographic.
The vast majority of Robinhood's customers aren't day trading clamoring to get their daily dose of tendies. The proportion of RH's customers who are actually effected is probably quite minimal, and of those some might change to another broker, but most will probably just put up with it. The market isn't going to be this active for long, and so once it calms down RH will put in place some fixes for its systems. This will be forgotten after a couple of weeks, it'll perform better next time and it'll have almost no impact on RH's value.
See my earlier posts about staying away from RH.
However, engineering HEADS SHOULD ROLL. In the era of on demand computing, near limitless cloud resources, etc, these engineering teams should have already had capacity planning ready to go for these trading platforms. I'm extremely disappointed - these engineers LIVE for this stuff, and they dropped the ball.
Again, heads should roll. This is not some data center that caught fire, this is capacity planning at its lowest common denominator.
I don't know if it was the case for the two other platforms OP mentioned but Robinhood seems to have some issues handling the request volumes.
Should be pointed out that BoA decided to build their own "cloud"
1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely.
2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info?
I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.
https://topstonks.com/blog?post_id=robinhood_down_for_the_co...
Needless to say, people are definitely considering moving to other brokers.