But it could also be the case that a service they use times out if you ask for invalid data e.g. from the future. In which case that could cause a cascade of backlogged tasks that would eventually lead to 503 errors under load.
The numbers can be totally wrong without being out of range. How many days was it calculating until March 1st, and how was it handling the next day which should have been +1, which surprisingly (to the code) wasn't March 1st? How many business hours passed from sun up Feb 28 until sun down March 1? Which day was the following Monday? No over- or underflows needed here, but clearly coding without taking these into account will provide something other than the correct answer.
As a guy who works in infosec since the early 90's, I see these errors showing up a lot in different, almost (anti)ingenious ways. This Robinhood event is just one for the collection, but not one that surprises me or anyone else in the field given enough time or enough education.
Fun fact: Y2K was a leap year. And we survived both events back-to-back better than Robinhood survived the obvious singular event in 2020. That is the problem here. Not Leap Year. But the fact that some programmers refuse to learn from history. At least during the Y2K debacle, everyone understood what they needed to do to code for leap years.
I used to teach the Easter Problem in a certain big blue coding classroom: From scratch, calculate the Easter dates for any given AD year. It gets messy quickly and it is not trivial. You have to think about Leap Years and Y2K's, which is why I was teaching it to rooms full of technical folk that hopefully remember it. Of course given enough classroom discussion, everyone got to some sort of code that predicts Easter for any given year, but not after quite a few failures. Then we talked about leap seconds, although those were a lot less painful. By that point, they totally understood how barely stable our calendar cycles are.
It proved a good lesson that I thought was rather important once they already understood the counting-fenceposts issue, which is yet another failure to understand cycles in exactly the same way. Needless to say, nobody at Robinhood was in my classes as far as I can tell. It's not like we don't have these "changes everything" roll overs fairly often.
Anyway, my job was to react if the bank I was working for ended up with crashing systems due to viruses, worms and whatever other malware they might have that weren't Y2K compatible. So this subject is near and dear to me. I'm literally blown away that Robin Hood and other companies could still have this problem today.
The last leap year was, gee, how many years ago? They didn't see that coming?
I might believe the response from RH if
1) this didn't happen exactly 4 years ago
2) admission of this being the issue didn't expose them to a negligence lawsuit which could easily bankrupt the company
3) the original tweet didn't have a screenshot of the app building links to 3/3/2020 data on 3/2/2020
Fair enough, but the screenshot that one client posted definitely shows their links generating dates for 3/3 on 3/2. That's highly suspicious in of itself. Also, the fact that their website previously said they had identified the cause and now it doesn't is also suspicious to me. I'm not a client so I don't care either way. I'm just saying I don't inherently believe a word RH says when they have so much to lose.
I notice that the status.robinhood.com page has changed as well. Yesterday there was a note about the source of the issue being identified. That is now gone.
12 comments
[ 3.0 ms ] story [ 30.1 ms ] threadCorrelation is not causation, but it is a big old spotlight that says "Hey, have a look over here!"
As a guy who works in infosec since the early 90's, I see these errors showing up a lot in different, almost (anti)ingenious ways. This Robinhood event is just one for the collection, but not one that surprises me or anyone else in the field given enough time or enough education.
Fun fact: Y2K was a leap year. And we survived both events back-to-back better than Robinhood survived the obvious singular event in 2020. That is the problem here. Not Leap Year. But the fact that some programmers refuse to learn from history. At least during the Y2K debacle, everyone understood what they needed to do to code for leap years.
I used to teach the Easter Problem in a certain big blue coding classroom: From scratch, calculate the Easter dates for any given AD year. It gets messy quickly and it is not trivial. You have to think about Leap Years and Y2K's, which is why I was teaching it to rooms full of technical folk that hopefully remember it. Of course given enough classroom discussion, everyone got to some sort of code that predicts Easter for any given year, but not after quite a few failures. Then we talked about leap seconds, although those were a lot less painful. By that point, they totally understood how barely stable our calendar cycles are.
It proved a good lesson that I thought was rather important once they already understood the counting-fenceposts issue, which is yet another failure to understand cycles in exactly the same way. Needless to say, nobody at Robinhood was in my classes as far as I can tell. It's not like we don't have these "changes everything" roll overs fairly often.
Anyway, my job was to react if the bank I was working for ended up with crashing systems due to viruses, worms and whatever other malware they might have that weren't Y2K compatible. So this subject is near and dear to me. I'm literally blown away that Robin Hood and other companies could still have this problem today.
The last leap year was, gee, how many years ago? They didn't see that coming?
https://twitter.com/AskRobinhood/status/1234861941413351434
March 1st, 2016 was the leap year. Not March 2nd.
"March 2nd 2016 was on a Tuesday though. Wouldn't this have happened on March 1st 2016 if this was the case?"
"It would have. Notice how none of the replies 4 years ago mention anything about a leap year."