Short version: He didn't fuel Raikkonen's car up enough which forced an early pit stop in the second-to-last lap of the race. However at the end of the race they placed the same as they had planned, and they had for the laps before the pit stop the advantage of having a stress-free race due to being in pole position.
And the lesson he learned from that: Don't stick to the rules too closely, but also try to see if breaking some rules can have positive side effects. Experiment.
Not a fun read. Very badly written as he constantly leaves out the consequences and details of decisions and circumstances, or only mentions them paragraphs later.
The lesson wasn't "Don't stick to the rules", it was "Capitalise on your mistakes". He didn't plan to underfuel the car, but when he did there were lessons to be learned.
It was overall a pretty terrible read. Likely because he's not allowed to share full details of what exactly happened.
Running a light fuel load in quali was a well-known strategy, and it's one they would have modeled in simulations. Likewise, avoiding first-lap (and particularly first-corner) collisions by qualifying as high up as possible is similarly a well-known strategy. Finally, they obviously know the money they get from being on pole.
The article doesn't bother explaining why the real-world outcome was different from what was modeled, so it's hard to gain any insight from it.
I'd be interested if anyone has any anecdotes that compare from software engineering. Sending a car out with suspension thingamies still attached seems like the equivalent of pushing a build to production with `DEBUG = True`, and I'm not sure much useful can be learned by you in that situation.
If you don't learn something from pushing a debug build to production, your debug build isn't instrumented enough. Maybe all you learn is what the effect of building with/without optimisation is on the real workload.
There's practically an entire genre of "here's what I learned from my expensive mistake" stories. Usually they teach humility and caution. This one is interesting because it's a variant of YAGNI. Imagine that you've carefully tuned your (software/car) for a specific set of parameters, then throw one out of whack by mistake - does this get caught by monitoring? Do you even notice? If you hillclimb from the new status to an optimal one, do you get back to the same local optimum?
Then of course there's the famous RISKS digest, which can be funny or depressing depending on your mood and dependence on complex systems.
(My own personal one of these? "killall" behaves differently on Solaris than it does on Linux ..)
I'd also strongly suggest not typing "exit" into kdb, the Solaris debugger when it is in write mode. It sets the global pointer for exit (the libc function) to NULL and the whole system will crash a few seconds to minutes later when the next process exits.
(It's not uncommon to use this tool to reconfigure some configuration values at runtime)
At least not on Solaris 10 or any of the open source derivatives. Seems it was special cased at some point in Solaris 11.
Had to make that mistake 2-3 times before I finally realised what I was doing to crash it given the slight delay.
You could learn from 'DEBUG=True' that you haven't automated your production deploy process carefully enough, including but not limited to testing failure cases. Raising the priority on that and making it part of routine work can save you from innumerable costly boneheaded mistaken deploys in the future.
I understand this, I guess what I'm trying to get at is that the learning from the article was more than just procedural (I.e how can we make sure this doesn't happen again), and more about the system itself.
Particularities elided, I was working on a POC of pretty critical (possible damage) closed loop (tightly controlled by operator at the time) system - basically read inputs, react according to algorithm, rinse and repeat. Inputs, however, came from multiple unsynchronised sources and we could arrive at situations where output depended on multiple inputs. I guess we took pretty natural decision to use latest available value of inputs when needed.
During one test run, one input did not update, we were happily feeding stale input to algorithm, which in turn took control paths that could have caused severe damages in unsupervised environment. Lessons learned: if you rely on invariants about inputs in code (update interval in this case), guard against violations of those invariants. This helped me many times later.
Not necessarily. Very often we extract some invariants from data and embed them in code because it makes code much cleaner, easier to write, etc.. The problem is that some things that we assume (sometimes explicitly, sometimes by mistake) to be invariants are not necessarily invariants after all. Very, very unlikely to change, but not invariants nevertheless.
I guess rather well-known real world example is systems starting to show weird behaviour when database resultset ordering changes (e.g. changes in indexing) and someone relies on correct entry being first/last.
Assumed invariant here is resultset ordering and code is written with this particular invariant in mind. If not enough ordering clauses are used, resultset ordering is not invariant, even though it worked for years already :)
We broke one API for 2 months, and thereby discovered that no-one was actually using it and so removed it.
We left some database healthchecks running on every request for a while, making load times a factor of 10 longer than they needed to be. Learnt our stuff was really fast and we didn't need to worry about performance much, I guess.
We occasionally ship a build with the equivalent of 'DEBUG=true' to (parts of) production. Deliberately. For an extremely short time period.
What that gains us is a run with real data and a build that's instrumented with assertions - which flushes out a lot of previously unknown issues.
So yes, you learn useful things. If you ever did it accidentally, the useful thing you learn is that shipping ideally is not an on/off switch, but a carefully monitored process :)
That's fascinating. Is there any reason to do this in production vs. a staging environment (assuming you've got one prod-like enough with replayed traffic)? It seems a little risky to me!
It's sort-of a staging, sort-of production environment :)
I'm working on desktop software, and this happens to builds that are full production builds, but that don't go to stable. So a very small subset of our users - who has agreed to being guinea pigs - occasionally gets these.
It also helps that we have a clear policy that assertions can only fire in cases that truly would be a bug in the wild - you can't assert and handle the issue. Either you think it might happen, then you handle it - or you state an invariant, in which case there's no reason to handle it being false. In other words, the only price you pay is that your build is slower - any assertion that fires would've been a silent bug before anyways. And, to be as close to non-debug as possible, any assertion firing merely creates a crash dump, it doesn't actually crash the executable.
And finally, we simply need to do it in the wild because many of our bugs are simply due to the staggering number of environments desktop software encounters.
The story was interesting but the conclusion was basically "and that was my long winded way of telling you to learn from your mistakes" which was pretty meh IMO but it got me thinking.
I wonder how long it will be before AI can take common generic advice such as "learn from your mistakes" and data about someone and generate a coherent article like this one.
"Learn from your mistakes" is an over simplification. Typically that means "if you make a mistake, figure out a way to not make it again". Instead, the lesson here is to find opportunities in the outcomes of your mistake. Additionally, build that philosophy into your organization.
There is quite a bit of research that implies that the most influential action you can take to make a software team more productive in terms of both quality and quantity is to build a culture that is a close sibling to what he describes.
Add in a little more of a socially supportive environment into the professional mindset depicted in the article to adapt it to software develop ent.
Psychosocial well-being is essentially dependent on social support, and it has been shown to improve creative problem solving. Sometimes improving performance to such a degree that it makes it hard to believe.
> There is quite a bit of research that implies that the most influential action you can take to make a software team more productive in terms of both quality and quantity is to build a culture that is a close sibling to what he describes.
Can you give us some pointers to said research? I'd love to see the details.
Google's Project Aristotle came to similar conclusions, you might look into that.
The rough TL;DR is good teams have all members speak up when necessary and good psychological safety, defined as the feeling that you can be wrong and you wont be rejected, punished or embarrassed for it.
Over the years I've done R&D as well as manufacturing engineering. The closer you are to the manufacturing end the tighter the processes are nailed down, you should still learn for your mistakes but at all times have a controlled default. I'd say the F1 race is like that as well, if the low fuel case was not tried before, which surprises me, it should be a wake up call for the upstream R&D guys to better cover their process space. For the guys on race day it's a sign that process monitoring needs to be improved. Anyway, everyone should benefit in the end and most places I've been have had this attitude. When it comes to learning from your mistakes individuals have a wide set of reactions, though, and it's sometimes tough to swallow your pride and debug the problem in front of you instead of quickly moving on and trying again.
I don't read it as this wasn't tested before in R&D, but perhaps it was and gave no advantage, so was not considered. The mistake allowed them to look for other positive aspects that weren't considered before:taking the pole in circuits where the first corner is complicated and messy can be a big advantage (less risk of collision and ending the race in the same position is something good). Also making all those companies who pay a lot of money to put stickers on your car a little happier.
> before the race weekend, all our simulations, which had been firmly plotted around a more conventional qualifying strategy, had told us we were likely to finish third.
That was 2006. Do anybody have information about to the software racing teams use to simulate race strategies and how many GPs they test? Thanks.
Big Monte Carlo simulations, and I'd expect them to test every GP. Based on where they qualify they have pre-computed decisions about when to pit / change tire types, etc. under various conditions, like a safety car on lap 10, etc. See: http://www.canopysimulations.com/. Not sure how many of the teams use that specific software, but the computing side of F1 is actually a big deal: the rules place not only place limits on the car design, but on the teraflops used to design it.
Which is actually true! The next best thing you can do at that track besides getting pole is to time the stops perfectly to maximize off-track passing, and that's what the simulations generally focus on.
I didn't know that there were mandated limits on processing power. Do you have any more info on this?
What processing are there limits on? Is it just the wind tunnel equivalence (Were there limits on wind tunnel time back in the day?)? Is it just pre-manufacturing simulations or do they limit the sort race simulations that we're talking about here?
The limit for the last couple of years has been 25 Teraflops at 64-bit precision. They have to approve and monitor the specific computing resources as well, and the mandatory shutdown periods during the year apply to said resources. This applies to computational fluid dynamics simulations - I have not heard of restrictions on strategy simulations. See appendix 8 in the sporting regulations: http://www.fia.com/regulation/category/110.
We were not allowed to tell anyone. You have to ask the teams directly, but they also won't reveal their secrets. I would have so many interesting F1 stories...
I was expecting an error considerably larger than a fuel calculation which ended up giving a better race strategy in terms of surviving the first corner, and netted the same position overall as predicted! I used to prepare my own rally car (up to WRC level), and made many mistakes much bigger than that, but I guess that explains why he works for an F1 team and I'm currently looking for job!
What did you learn from those mistakes? Did you learn how to not make them again or possibly even something else beneficial?
I think you might have missed the point of the article. Excellence is not in avoiding mistakes, Excellence is learning from mistakes so you can do better next time.
Or you can do what we do at BiglyCorp, have an absolute shitshow release where 250,000 customers crashed. Then act as if nothing happened and go along doing the same things that caused the shitshow. If you don't learn from mistakes, you've optimized for mistakes.
Could be worse. I still wake up sweating thinking about a former job where several million identities were stolen. Tech staff tried to prevent it, even predicting high risk of mistake, but management pushed through it anyway.
A business mess is one thing, but losing somebody's identity is another level of feeling shitty.
At first glance, this might seem like "learn from your mistakes". But after thinking about it, this article is much more.
Put simply, nothing will ever go perfectly right, and nothing will ever go perfectly wrong. In any situation, there will be things that you want to see happen again, and things you'd like to get rid of.
In Formula 1, that probably happens often. This is an engineer's take on what that's like in one of the most demanding sports (business) on the planet.
Yes it locks it in place so the car doesn't move during transport. F1 cars derive so much of their suspension movement from the flexing of the tyre wall that the car is still drivable.
Interesting the local f1 team RBR (red bull) a couple of years ago had two DNF's from not correctly managing the fuel load.
I did do a spec job application always wondered if I would have pointed out that as a mistake that I would have not made more than once :-) BTW my first job was at the worlds leading rnd organisations in fluid dynamics which is also nearby on campus at cranfield
Can someone explain this to me? I'm only familiar with the current regulations and so the logic of this paragraph doesn't make sense to me:
> Back in 2006, the regulations required teams to qualify their cars with race fuel-loads. The trade-offs were obvious – load the car up with fuel and you’d end up further down the grid, but with the benefit and flexibility of adjusting your strategy in the race. By contrast, qualify on petrol fumes and you had a good chance to be quick on Saturday, but you’d theoretically start the race with one hand tied behind your back.
There's a tradeoff to be made with amount of fuel loaded into the car. If you put more fuel in the car, it weighs more, and so will corner and accelerate worse and have worse lap times. On the other hand, it'll have more fuel, so it won't need to make a pit stop as soon.
Qualifying laps are structured differently from the actual race itself, in a way that incentivizes putting less fuel in the car. Essentially, only the qualifying lap times matter, and not the advantages from having more fuel loaded. The obvious strategy is to put the bare minimum in the tank for qualifying, and more during the start of the actual race.
The new regulations forbid that strategy - you're allowed no more fuel to start the race than you had used to start the qualifying laps.
In the last years, no refueling stop were allowed during the race, so everyone just loaded the same amount of fuel for the race.
However, previously, there was refueling possible during the race, which allowed for some strategy variation:
- either you start with small amount of fuel, gain advantage due to lighter car (~1s faster per lap), and refuel early in the race (probably 2 refueling stops needed in such case)
- or start with bigger amount of fuel and stay longer on the track, but only do 1 stop for refueling
Also, since you needed to start the race and qualifications with same amount of fuel, going with less-fuel-and-double-refuel strategy meant you were more likely to win a pole position due to lighter car during qualifications.
69 comments
[ 3.4 ms ] story [ 128 ms ] threadedit --
It's some Javascript based redirect, I can see this in curl: ... "baseRedirect":"https://www.linkedin.com/uas/login?session_redirect="}--> ...
I'm not investigating further.
And the lesson he learned from that: Don't stick to the rules too closely, but also try to see if breaking some rules can have positive side effects. Experiment.
Not a fun read. Very badly written as he constantly leaves out the consequences and details of decisions and circumstances, or only mentions them paragraphs later.
Running a light fuel load in quali was a well-known strategy, and it's one they would have modeled in simulations. Likewise, avoiding first-lap (and particularly first-corner) collisions by qualifying as high up as possible is similarly a well-known strategy. Finally, they obviously know the money they get from being on pole.
The article doesn't bother explaining why the real-world outcome was different from what was modeled, so it's hard to gain any insight from it.
There's practically an entire genre of "here's what I learned from my expensive mistake" stories. Usually they teach humility and caution. This one is interesting because it's a variant of YAGNI. Imagine that you've carefully tuned your (software/car) for a specific set of parameters, then throw one out of whack by mistake - does this get caught by monitoring? Do you even notice? If you hillclimb from the new status to an optimal one, do you get back to the same local optimum?
Then of course there's the famous RISKS digest, which can be funny or depressing depending on your mood and dependence on complex systems.
(My own personal one of these? "killall" behaves differently on Solaris than it does on Linux ..)
(It's not uncommon to use this tool to reconfigure some configuration values at runtime)
At least not on Solaris 10 or any of the open source derivatives. Seems it was special cased at some point in Solaris 11.
Had to make that mistake 2-3 times before I finally realised what I was doing to crash it given the slight delay.
Tiny example, GNU ls:
ls -lh bla : ok
ls bla -lh : ok
Most non-GNU lses, second form : oh-oh! :)
During one test run, one input did not update, we were happily feeding stale input to algorithm, which in turn took control paths that could have caused severe damages in unsupervised environment. Lessons learned: if you rely on invariants about inputs in code (update interval in this case), guard against violations of those invariants. This helped me many times later.
Every time you use assert() somewhere you're doing just that.
I guess rather well-known real world example is systems starting to show weird behaviour when database resultset ordering changes (e.g. changes in indexing) and someone relies on correct entry being first/last.
Assumed invariant here is resultset ordering and code is written with this particular invariant in mind. If not enough ordering clauses are used, resultset ordering is not invariant, even though it worked for years already :)
We left some database healthchecks running on every request for a while, making load times a factor of 10 longer than they needed to be. Learnt our stuff was really fast and we didn't need to worry about performance much, I guess.
What that gains us is a run with real data and a build that's instrumented with assertions - which flushes out a lot of previously unknown issues.
So yes, you learn useful things. If you ever did it accidentally, the useful thing you learn is that shipping ideally is not an on/off switch, but a carefully monitored process :)
I'm working on desktop software, and this happens to builds that are full production builds, but that don't go to stable. So a very small subset of our users - who has agreed to being guinea pigs - occasionally gets these.
It also helps that we have a clear policy that assertions can only fire in cases that truly would be a bug in the wild - you can't assert and handle the issue. Either you think it might happen, then you handle it - or you state an invariant, in which case there's no reason to handle it being false. In other words, the only price you pay is that your build is slower - any assertion that fires would've been a silent bug before anyways. And, to be as close to non-debug as possible, any assertion firing merely creates a crash dump, it doesn't actually crash the executable.
And finally, we simply need to do it in the wild because many of our bugs are simply due to the staggering number of environments desktop software encounters.
I wonder how long it will be before AI can take common generic advice such as "learn from your mistakes" and data about someone and generate a coherent article like this one.
Add in a little more of a socially supportive environment into the professional mindset depicted in the article to adapt it to software develop ent.
Psychosocial well-being is essentially dependent on social support, and it has been shown to improve creative problem solving. Sometimes improving performance to such a degree that it makes it hard to believe.
Can you give us some pointers to said research? I'd love to see the details.
The rough TL;DR is good teams have all members speak up when necessary and good psychological safety, defined as the feeling that you can be wrong and you wont be rejected, punished or embarrassed for it.
That was 2006. Do anybody have information about to the software racing teams use to simulate race strategies and how many GPs they test? Thanks.
What processing are there limits on? Is it just the wind tunnel equivalence (Were there limits on wind tunnel time back in the day?)? Is it just pre-manufacturing simulations or do they limit the sort race simulations that we're talking about here?
I think you might have missed the point of the article. Excellence is not in avoiding mistakes, Excellence is learning from mistakes so you can do better next time.
A business mess is one thing, but losing somebody's identity is another level of feeling shitty.
Put simply, nothing will ever go perfectly right, and nothing will ever go perfectly wrong. In any situation, there will be things that you want to see happen again, and things you'd like to get rid of.
In Formula 1, that probably happens often. This is an engineer's take on what that's like in one of the most demanding sports (business) on the planet.
What is a set-up packer?
I did do a spec job application always wondered if I would have pointed out that as a mistake that I would have not made more than once :-) BTW my first job was at the worlds leading rnd organisations in fluid dynamics which is also nearby on campus at cranfield
> Back in 2006, the regulations required teams to qualify their cars with race fuel-loads. The trade-offs were obvious – load the car up with fuel and you’d end up further down the grid, but with the benefit and flexibility of adjusting your strategy in the race. By contrast, qualify on petrol fumes and you had a good chance to be quick on Saturday, but you’d theoretically start the race with one hand tied behind your back.
Qualifying laps are structured differently from the actual race itself, in a way that incentivizes putting less fuel in the car. Essentially, only the qualifying lap times matter, and not the advantages from having more fuel loaded. The obvious strategy is to put the bare minimum in the tank for qualifying, and more during the start of the actual race.
The new regulations forbid that strategy - you're allowed no more fuel to start the race than you had used to start the qualifying laps.
However, previously, there was refueling possible during the race, which allowed for some strategy variation:
- either you start with small amount of fuel, gain advantage due to lighter car (~1s faster per lap), and refuel early in the race (probably 2 refueling stops needed in such case)
- or start with bigger amount of fuel and stay longer on the track, but only do 1 stop for refueling
Also, since you needed to start the race and qualifications with same amount of fuel, going with less-fuel-and-double-refuel strategy meant you were more likely to win a pole position due to lighter car during qualifications.