This is a good talk. Really gets into the details of how things differ from the classical SaaS or consumer product.
I've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :)
One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but when you're running a global consumer product, it's a lot harder to do that without pushback.
So for most of us, our stress is around zero downtime maintenance, and theirs is around never dropping a request when the system is live.
Not sure what the practical difference is (24/7 vs ~10/5) except for the convenience when planning data migrations if you have regularly planned downtime.
For most code changes being turned off at night isn't much of an advantage, as the new code will need to go live at some point and that point is where the risk is. For systems on 24/7 you simply need a copy of your production environment to test on, a.k.a. staging.
The main thing about 24/7 is needing follow-sun SRE and/or out of hours oncall.
I love talks like this so much. Trading isn't something I hardly ever think about as I live/work in a bubble of sorts, and it was a fascinating listen.
I have heard similar talks from Shopify and such back in the day, about their own product, but always love listening to more.
Nope. Jane Street engineers just love what they do; and do it for fun in their own time and professionally and they love solving puzzles.
They just think for themselves to make money and not what their manager tells them to do unless either the manager or trader believes they will lose money and the game.
The only relevant difference to you is that you must have zero morals or ethics in this game and it is actually a high stress environment behind the videos and engineering propaganda which is the slender difference between beating or losing to other competitors for their clients.
So really stop worshipping them and fuelling their egos as that is what they (Jane Street) want you to do.
9 comments
[ 3.1 ms ] story [ 29.1 ms ] threadI've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :)
One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but when you're running a global consumer product, it's a lot harder to do that without pushback.
So for most of us, our stress is around zero downtime maintenance, and theirs is around never dropping a request when the system is live.
For most code changes being turned off at night isn't much of an advantage, as the new code will need to go live at some point and that point is where the risk is. For systems on 24/7 you simply need a copy of your production environment to test on, a.k.a. staging.
The main thing about 24/7 is needing follow-sun SRE and/or out of hours oncall.
I have heard similar talks from Shopify and such back in the day, about their own product, but always love listening to more.
They just think for themselves to make money and not what their manager tells them to do unless either the manager or trader believes they will lose money and the game.
The only relevant difference to you is that you must have zero morals or ethics in this game and it is actually a high stress environment behind the videos and engineering propaganda which is the slender difference between beating or losing to other competitors for their clients.
So really stop worshipping them and fuelling their egos as that is what they (Jane Street) want you to do.
The clickbait title of "billions of dollars a day" is nothing to praise.