"Next, we eliminate some incorrect worlds by testing them against noisy data. This data should come from intermediate steps and can’t be quantified. If the data conflicts with an imagined future, that future is unlikely to occur. Noisy information may be bad for feedback loops, but it’s great for elimination, because the wider range of outcomes is more effective for pruning. It may even be a bit faster than accurate data, since the variance of accurate data is lower!
As we near decision time, we stop the world construction and elimination, and use judgment and experience to pick the best idea out of the remaining ones. Process improvement happens by creating many parallel processes and discarding the ones that are unfit, not by iteratively improving a single process."
The author gives an example of this afterwards (analysing shares in a pachinko company).
Essentially, you identify numerous hypothetical scenarios, analyse the impact they would have, and use that information to decide whether a particular course of action is a good idea or not.
I read the article and the example - I don't see it as an example of what he lays out. As you say - he laid out a bunch of questions and answered them. How is that testing anything against noisy data? Why are they intermediate steps? Where is the feedback loop? What if his answers are wrong and/or what if his sense for how likely they are is wrong?
What he calls "world construction" involves the development of a rubric custom to the problem.
This creates a faster feedback loop inside of the larger, noisier one. Your feedback is now guided around the question of "what makes the rubric itself better?" This can be done on principle, with limited access to external information. Philosophical thinking is eminently suited to this style of problem, but it can be supplemented with short-term empirical studies that add some falsifying points and narrow your cone of uncertainty.
At the end you've generated a list of yes/no questions forming the rubric of whether the course of action is likely to succeed. It can be turned into a ranking score, or a pass-fail threshold.
If you're frustrated by the idea of just making it up on principle, that's a frustration with philosophy itself; it rarely "works" until you accept some pragmatic premises around what is "good" or "true". The point of having a large number of questions, using a wide variety of perspectives, is that they test the overall coherency of the premise. Something can work fine from one perspective, and then completely fail in another. When that happens, it's a good sign that you have more to improve.
It's quite an important life skill to practice. It's easy to go along with the crowd, but this is a way of breaking away from it.
Not so sure about this: "An example of a gentle environment is software development. Outcomes are clearly specified, frameworks are common and work well, and your actions receive instant and accurate feedback."
Software development is a highly people-driven environment. You can introduce something in your process (e. g. a new framework) but the feedback comes only a year later when it becomes evident that the framework hasn’t been a good fit or was introduced poorly (e. g. team members are quitting, productivity has plummeted, tech debt has piled up).
The writer almost certainly isn't a software engineer. It's one of those things where you don't know how something works so you imagine the process as much simpler/less messy/more well defined than it really is...
A year is still blazingly fast. Most plumbers will wait years to try a new way of joining pipes, and it might take decades until it's clear if the new way is more reliable.
In the time it takes most professions to consider trying a new framework or tool (beyond the early adopters), entire programming languages rise to popularity and fall back into obscurity.
I see what you say. Although I'm not the author I will try to speculate on what they might mean by "instant and accurate feedback":
I heard the same sentiment from an architect friend. (The kind of architect who designs buildings) He said that in their business you put your best ideas on paper. Most plans then don't get built for reasons outside of your control. Projects fail, companies fold. Sometimes your designs can't pass review with the authorities because you misunderstood the regulations. Sometimes it can't pass review because the local government doesn't like your customer. Years down the road some of your plans are built. If there are problems with the design some problems might be obvious immediately, but most will only show up as the building ages. And if there are problems, are they there because you designed it wrong? Or the builders skimped on materials? Or is it maybe because the costumer meddled with your designs? You will never know. The feedback loop is both years long, and the execution is inaccurate.
On the other hand besides being an architect he was also dabbling in hobby programming projects. There if he made a syntax error the computer pointed it out to him within seconds. And even when he found a bug after compilation he could be certain it wasn't because the compiler saved money by using 10% less screws than specified or something. That is to say the feedback was both fast and accurate. If you had bugs, most probably it was because you made a mistake.
Indeed, the opposite of all three assertions is true in practically every project I ever worked on. Not sure if the author is joking, or simply has no software dev experience.
Yes, and I understood his point because, as a business-minded software engineer, I can map the edit-run-test-debug process to what he calls a tight loop (the Maths Test).
I also recognise the fuzzier process of designing features with stakeholders, implementing them, deploying them and having them tested in the market as something between the Rocket Launch (for the technical process from ideation to deployment and subsequent maintenance) and Stock Picking (for the product/market fit).
To finish populating the 2x2 matrix, the Novice Bowler approach is more similar to what many of the students used to do back when I was tutoring Algorithms and Data Structures at uni. They'd throw things at the IDE until it compiled, and call it a day.
Tight feedback loops are great for software development, but terrible for systems administration. In my career, I regularly have these conversations:
"We did A, it was slow, then we did B, and it was fast." == "Congratulations, you discovered that the cache works, not that B is faster than A."
"We ran this test on AWS image size such-and-such, and it performed 5% better than the competing product that we tested previously." == "That's just noisy neighbours on a multi-tenant platform, you've learned nothing."
"We ran a test and it succeeded, hence it is now working!" == "The system fails nine out of ten times randomly, you just stopped when you had your first success. Try it again."
This ability to spin up a test environment quickly and run it through a scenario can be intoxicating. People try all sorts of random stuff, see a "win", and then call it a day without really doing the legwork to prove that they achieved something meaningful.
This guy has had a few talks at CppCon over the years, and covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals: https://www.youtube.com/watch?v=koTf7u0v41o
In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :)
Simple methods I use and can recommend are:
- Ideally, collect data from production, where the change behaves exactly as it would in production because it is production!
- Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc...
- Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc...
- Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance.
- Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph.
- Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph above can help, but only if your input size goes waaaay past the biggest cache sizes in the system.
As a real-world example: I recently reduced the size of a database platform by a few vCPUs to save on licensing costs, but increased the per-core performance and simultaneously did some network tuning. Some benchmarks showed the same performance, others showed an improvement. Both were extremely noisy before and after the change.
I would like to take credit and say that I improved performance while saving money. However, the scientifically honest result is that any performance change is statistically insignificant.
The cost saving is significant, because it has very little variance. That's good enough, and I don't have to make a statement about performance that the numbers can't really back up.
I would add always (for network servers) run “open” loads tests with each request being independent of the fates of the other requests, probably from a Poisson distribution (generated by having the delays be pulled from exponential random distribution ( https://en.wikipedia.org/wiki/Exponential_distribution ).
So many load tools send a request then wait for the reply then send the next. That is basically useless for understanding variance under spiky load and for assessing when a system is below its capacity how stable it is.
For real world performance, the variance matters a lot. The code will make some things be cached some not (so the data variability in the test load should follow the same sort of power law that your real traffic does), and even if you get 10% faster at the median if the variance goes up the quality is lower.
Any part of statistics is completely dependent on calculus. You don't get to choose how features are distributed over the set of possible samples. It's always some scary looking function without exact solutions.
But you don't need to teach people all the fundamentals of calculus before they can apply it. Definitions and a small set of pre-made answers are enough to start using it and focusing on the statistics. When (and if) those people decide to learn it more in depth, they can go back to calculus fundamentals, and will have an easier time doing so, because they will know why those things exist.
Basic probability (coin flipping, events over trials) by itself is just as useless as basic trigonometry.
It only becomes useful as you move from simple combinatorics to continuous distributions, at which point you need differentiation and integration to switch between CDF and PDF. Much in the same way trigonometry becomes useful in physics once you need to deal with vectors and waves.
We teach maths too much, we should just give gold bars to every student so they already win at life.
Trigonometry in particular is so ingrained in many aspect of applied science that not learning its fundamentals and then pushing to its higher level concepts is going to seriously reduce adaptation capability in students.
And if you somehow never need to rotate things, it's a good training in applying theoretical rules to concrete problems. In my university, after 2 years of increasingly complex trigonometry, we were left with 3D engine programming and to the fact that opened my mind forever that theoretical mathematics have a purpose.
What % of students struggle and never really get the maths? What % do fine with the maths but never need them in their adult lives? I would argue this is far more than 75% of all people. Same can be argued probably with higher %s for foreign language in the us. I think learning advanced maths and foreign languages are great but I also think we waste insane amounts of human potential and $$$ with the way we go about it.
I disagree, tight feedback loops are always better in every scenario than loose ones. What you do with the new information matters though.
E.g. you would always choose to read an accelerometer sensor in a tight loop in a flight control system - however, you wouldn’t just blindly apply the latest reading as an input to your system (accelerometers are extremely noisy, the data looks like complete trash on the front end before applying some kind of averaging filter).
Further down the chain on a flight control computer is also a good counterpoint to this idea that you should prefer a more lossy sample rate.
Control theory is an established field of study. Better to adopt filtering & PID control strategies than to under sample an input.
Yeah, it is pretty wooly. How about taking it from the opposite position, almost like a proof by contradiction, e.g. the opposite position might be “you can see better if you look less often”.
I feel like you're conflating "sampling rate" with "tight feedback loop". Yes, a higher sampling rate (assuming each sample remains at or above the level of quality of your lower rate samples) is always better. But a tighter feedback loop is one where you are making decisions with that data at a faster clip too. As pointed out in the article (and by yourself), this is not always good. If you are using some averaging filter, as you say, then that is being "looser" about your feedback loop. You are recognizing that just because the accelerometer is telling you that you are going down doesn't mean you are about to crash. Someone with an overly tight feedback loop would not be so careful, and might immediately course-correct as soon as they sense they are going down unintentionally.
The article is talking about feedback loops in terms of learning, not "business-marketing-whatever bullshit advice" and I think the author makes this clear.
This paragraph sums up the motivation of the article quite nicely:
"But after achieving proficiency in a field, tight feedback loops are useless. That’s because initially, the learning environment is gentle. The path is well-travelled, there are easily accessible guides, things work according to common sense. As we become proficient, the environment becomes harsher, with noisier feedback. Improving is not as easy. At higher levels of skill, further progression depends on self-learning, on discovering or inventing new practices and knowledge. As skill increases, the gap between optimizing for metrics and optimizing for mastery widens."
It's better to adopt filtering instead of reducing your sample rate, only if the opportunity cost of sampling and processing is sufficiently cheap (given scenario). If it's relatively expensive to sample and process (which I think apply to many business type scenarios),, just decreasing your sampling rate to more appropriately match your ability to manipulate outcomes make sense.
The old destructive vs non destructive testing idea. We measured the screw but we destroyed the screw. It had good quality. Now, how many screws do we need to sacrifice to be on top of our quality?
Great stuff, in particular the part about arriving at a local maximum is something I've observed a lot as well.
I think I was introduced to the concept in some talk about AI, something which I never really pursued. Nevertheless, the concept stuck with me and I've been reminded of it professionally more than once.
I was right there with it until it got to ... the stock market.
I get it - we live in a money-oriented society.
But presenting a psychological learning model as useful as this one and limiting it to the stock market feels ... narrow minded. It sounds like "being a jedi is nice because you get this fancy tool with which you can cut AND braise sausages at the same time".
Use this to get more healthy. Use this to spend more and better time with your family and friends. Use this to live a more fulfilling life. And sure, use it for the stock market or your day job. But that could be a couple phrases at the end, not the "meat" of the thing.
Interesting take, I really like the 2x2 table that brings such clarity. Funnily, my mind immediately jumped to the question on how to convert the feedback loop types to a better one. Of course, the whole point of the article is that it is not always a good idea, but sometimes it may be, and even just being aware of the nature of the problem seems beneficial. Seems like a great addition to a mental model toolbox.
This was an attempt at a contrarian clickbait blogpost.
Obviously noise is a problem. That's a separate issue from loop tautness. Tight is a measure of loop size not accuracy of information as the article claims. This is similar to accuracy vs precision.
It's worth adding that whether or not the right thing is measured is not related to feedback loop size. The "timely" 737 Max example that was peppered in shows this. Clearly what was being measured was not related to the problems the plane had.
The unsubstantiated (not even anecdotal) claims based on the author's confusion (e.g. data-less diagram of masters preferring loose loops) seem to be the author taking his own advice. They're complete speculation, not even pretending to be attached to reality.
Yeah, the author is using "accuracy" when he means "precision" on many occasions, and the lack of awareness of this distinction greatly lowers the blogpost's quality.
Attributing 737 MAX issues to tight control loops also shows critical failure in understanding what happened with 737 MAX.
The technical details of failures in MCAS system weren't the problem. The problem was the decision chain that caused creation of MCAS system, and that one had nothing to do with tight or loose control loops, and everything to do with business incentives perversing safety process.
>>When quantifying things, people naturally focus on things that can easily be measured. Measuring the final result doesn’t provide enough quantitative data, so it’s tempting to include the data from intermediate steps. This is an attempt to shorten the feedback loop, and trying to shorten feedback loops is very dangerous in complex systems.
This is a really important point that the author sort of glosses over -- adding measurements, or tighter measurements must be done extremely carefully for two reasons: 1) the risk is 'looking for the key under the streetlamp because it'd better light', and 2) unintended side effects.
The first will cause data to just be basically irrelevant or even inverse to the desired signal, but people will chase it anyway because it is easy to measure. The second taps into the management phenomena of "you get the results that you measure", and create perverse incentives to do the wrong thing, either self-managing, learning, or managing an org.
Good article, much more worthwhile than I expected!
Also in many processes, there are things going on at various time scales. Tight feed back loops are good for higher frequency phenomena. Ignoring the slower phenomena can make you like the kids who see the water retreating, build a sandbox, and then come back in the evening to see it all underwater. So many places have all the log data, terabytes, maybe petabytes, but just for three months. What happened last year? When can we expect traffic to break this scaling assumption and need a refactor, this year or next? Impossible to know, unless you make your own copies of summary data and stay around for a few years.
I'm actually thinking that this is part of what is going on with all the cries of "Inflation!!" - compared to last years' numbers, when everything was running at about 40%, of course prices are rebounding, and even overcompensating on the high side - the real question is for how long (there was also a massive hole blown in the money supply, which needs to be backfilled).
It seems like his alternative (test many scenarios and prune) makes it possible to fool yourself through randomness a different way if you don’t account for the fact that you’re doing multiple comparisons [1].
I don’t think the author does a good job of explaining the actual problem.
The novice bowler example has nothing to do the tightness of the feedback loop. The issue is solely the statistical variation in the results.
The novice bowler sometimes bowls too far to the left, sometimes to the right. This isn’t a result of conscious action, rather a lack of control. When the bowler aims for the middle of the lane and the ball randomly goes to the left on one occasion, aiming to the right on the next shot will not fix the problem; it will make it worse.
Summarizing the real issue for the bowler: don’t make decisions based on random data. If you don’t know to what extent the data you’re looking at contains a random element, you should try to figure out how much before attempting a correction.
Lengthening the feedback loop in this example might involve obscuring the bowler’s vision of the pins once the ball has begun rolling and presenting a video of the results 10 minutes later. It’s hard for me to imagine how this could be anything other than detrimental.
Another issue with the article is use of the term “accurate feedback”. It’s conflating two ideas, accuracy of data collection and statistical variation of a system.
Back to the bowling example (sorry for harping on it), unless you frequently hallucinate and see the pins fall when they did not, the results in practice are very very accurate. It’s just that they describe a complex system with an initially unknown amount of variation. It’s only through having precise measurements that you can even quantify the variation.
50 comments
[ 3.6 ms ] story [ 92.4 ms ] thread"Next, we eliminate some incorrect worlds by testing them against noisy data. This data should come from intermediate steps and can’t be quantified. If the data conflicts with an imagined future, that future is unlikely to occur. Noisy information may be bad for feedback loops, but it’s great for elimination, because the wider range of outcomes is more effective for pruning. It may even be a bit faster than accurate data, since the variance of accurate data is lower!
As we near decision time, we stop the world construction and elimination, and use judgment and experience to pick the best idea out of the remaining ones. Process improvement happens by creating many parallel processes and discarding the ones that are unfit, not by iteratively improving a single process."
I can't quite understand what he is saying.
Essentially, you identify numerous hypothetical scenarios, analyse the impact they would have, and use that information to decide whether a particular course of action is a good idea or not.
This creates a faster feedback loop inside of the larger, noisier one. Your feedback is now guided around the question of "what makes the rubric itself better?" This can be done on principle, with limited access to external information. Philosophical thinking is eminently suited to this style of problem, but it can be supplemented with short-term empirical studies that add some falsifying points and narrow your cone of uncertainty.
At the end you've generated a list of yes/no questions forming the rubric of whether the course of action is likely to succeed. It can be turned into a ranking score, or a pass-fail threshold.
If you're frustrated by the idea of just making it up on principle, that's a frustration with philosophy itself; it rarely "works" until you accept some pragmatic premises around what is "good" or "true". The point of having a large number of questions, using a wide variety of perspectives, is that they test the overall coherency of the premise. Something can work fine from one perspective, and then completely fail in another. When that happens, it's a good sign that you have more to improve.
It's quite an important life skill to practice. It's easy to go along with the crowd, but this is a way of breaking away from it.
Software development is a highly people-driven environment. You can introduce something in your process (e. g. a new framework) but the feedback comes only a year later when it becomes evident that the framework hasn’t been a good fit or was introduced poorly (e. g. team members are quitting, productivity has plummeted, tech debt has piled up).
In the time it takes most professions to consider trying a new framework or tool (beyond the early adopters), entire programming languages rise to popularity and fall back into obscurity.
I heard the same sentiment from an architect friend. (The kind of architect who designs buildings) He said that in their business you put your best ideas on paper. Most plans then don't get built for reasons outside of your control. Projects fail, companies fold. Sometimes your designs can't pass review with the authorities because you misunderstood the regulations. Sometimes it can't pass review because the local government doesn't like your customer. Years down the road some of your plans are built. If there are problems with the design some problems might be obvious immediately, but most will only show up as the building ages. And if there are problems, are they there because you designed it wrong? Or the builders skimped on materials? Or is it maybe because the costumer meddled with your designs? You will never know. The feedback loop is both years long, and the execution is inaccurate.
On the other hand besides being an architect he was also dabbling in hobby programming projects. There if he made a syntax error the computer pointed it out to him within seconds. And even when he found a bug after compilation he could be certain it wasn't because the compiler saved money by using 10% less screws than specified or something. That is to say the feedback was both fast and accurate. If you had bugs, most probably it was because you made a mistake.
I also recognise the fuzzier process of designing features with stakeholders, implementing them, deploying them and having them tested in the market as something between the Rocket Launch (for the technical process from ideation to deployment and subsequent maintenance) and Stock Picking (for the product/market fit).
To finish populating the 2x2 matrix, the Novice Bowler approach is more similar to what many of the students used to do back when I was tutoring Algorithms and Data Structures at uni. They'd throw things at the IDE until it compiled, and call it a day.
"We did A, it was slow, then we did B, and it was fast." == "Congratulations, you discovered that the cache works, not that B is faster than A."
"We ran this test on AWS image size such-and-such, and it performed 5% better than the competing product that we tested previously." == "That's just noisy neighbours on a multi-tenant platform, you've learned nothing."
"We ran a test and it succeeded, hence it is now working!" == "The system fails nine out of ten times randomly, you just stopped when you had your first success. Try it again."
This ability to spin up a test environment quickly and run it through a scenario can be intoxicating. People try all sorts of random stuff, see a "win", and then call it a day without really doing the legwork to prove that they achieved something meaningful.
In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :)
Simple methods I use and can recommend are:
- Ideally, collect data from production, where the change behaves exactly as it would in production because it is production!
- Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc...
- Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc...
- Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance.
- Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph.
- Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph above can help, but only if your input size goes waaaay past the biggest cache sizes in the system.
As a real-world example: I recently reduced the size of a database platform by a few vCPUs to save on licensing costs, but increased the per-core performance and simultaneously did some network tuning. Some benchmarks showed the same performance, others showed an improvement. Both were extremely noisy before and after the change.
I would like to take credit and say that I improved performance while saving money. However, the scientifically honest result is that any performance change is statistically insignificant.
The cost saving is significant, because it has very little variance. That's good enough, and I don't have to make a statement about performance that the numbers can't really back up.
So many load tools send a request then wait for the reply then send the next. That is basically useless for understanding variance under spiky load and for assessing when a system is below its capacity how stable it is.
For real world performance, the variance matters a lot. The code will make some things be cached some not (so the data variability in the test load should follow the same sort of power law that your real traffic does), and even if you get 10% faster at the median if the variance goes up the quality is lower.
But you don't need to teach people all the fundamentals of calculus before they can apply it. Definitions and a small set of pre-made answers are enough to start using it and focusing on the statistics. When (and if) those people decide to learn it more in depth, they can go back to calculus fundamentals, and will have an easier time doing so, because they will know why those things exist.
If yes, then I agree with him. People should have more probability lessons at school than trigonometry or calculus.
It only becomes useful as you move from simple combinatorics to continuous distributions, at which point you need differentiation and integration to switch between CDF and PDF. Much in the same way trigonometry becomes useful in physics once you need to deal with vectors and waves.
Trigonometry in particular is so ingrained in many aspect of applied science that not learning its fundamentals and then pushing to its higher level concepts is going to seriously reduce adaptation capability in students.
And if you somehow never need to rotate things, it's a good training in applying theoretical rules to concrete problems. In my university, after 2 years of increasingly complex trigonometry, we were left with 3D engine programming and to the fact that opened my mind forever that theoretical mathematics have a purpose.
E.g. you would always choose to read an accelerometer sensor in a tight loop in a flight control system - however, you wouldn’t just blindly apply the latest reading as an input to your system (accelerometers are extremely noisy, the data looks like complete trash on the front end before applying some kind of averaging filter).
Further down the chain on a flight control computer is also a good counterpoint to this idea that you should prefer a more lossy sample rate.
Control theory is an established field of study. Better to adopt filtering & PID control strategies than to under sample an input.
Sounds a little unconvincing.
> applying some kind of averaging filter
How do you choose the size of your averaging window?
How do you make sure you don’t catch a systematic bias due to some unrelated event?
Control theory may help in some cases (flight control) but possibly less so in others (stock market).
Yeah, it is pretty wooly. How about taking it from the opposite position, almost like a proof by contradiction, e.g. the opposite position might be “you can see better if you look less often”.
In that example you can have excessively tight feedback loops where looking less often let’s you see better.
The article is talking about "feedback loops" in some handwavy business-marketing-whatever bullshit advice sense.
This paragraph sums up the motivation of the article quite nicely:
"But after achieving proficiency in a field, tight feedback loops are useless. That’s because initially, the learning environment is gentle. The path is well-travelled, there are easily accessible guides, things work according to common sense. As we become proficient, the environment becomes harsher, with noisier feedback. Improving is not as easy. At higher levels of skill, further progression depends on self-learning, on discovering or inventing new practices and knowledge. As skill increases, the gap between optimizing for metrics and optimizing for mastery widens."
i feel like we are conflating tight feedback loops with some other concept that needs to be teased out.
small bets can be divergent.
https://searchcio.techtarget.com/definition/OODA-loop
I think I was introduced to the concept in some talk about AI, something which I never really pursued. Nevertheless, the concept stuck with me and I've been reminded of it professionally more than once.
I get it - we live in a money-oriented society.
But presenting a psychological learning model as useful as this one and limiting it to the stock market feels ... narrow minded. It sounds like "being a jedi is nice because you get this fancy tool with which you can cut AND braise sausages at the same time".
Use this to get more healthy. Use this to spend more and better time with your family and friends. Use this to live a more fulfilling life. And sure, use it for the stock market or your day job. But that could be a couple phrases at the end, not the "meat" of the thing.
Obviously noise is a problem. That's a separate issue from loop tautness. Tight is a measure of loop size not accuracy of information as the article claims. This is similar to accuracy vs precision.
It's worth adding that whether or not the right thing is measured is not related to feedback loop size. The "timely" 737 Max example that was peppered in shows this. Clearly what was being measured was not related to the problems the plane had.
The unsubstantiated (not even anecdotal) claims based on the author's confusion (e.g. data-less diagram of masters preferring loose loops) seem to be the author taking his own advice. They're complete speculation, not even pretending to be attached to reality.
The technical details of failures in MCAS system weren't the problem. The problem was the decision chain that caused creation of MCAS system, and that one had nothing to do with tight or loose control loops, and everything to do with business incentives perversing safety process.
This is a really important point that the author sort of glosses over -- adding measurements, or tighter measurements must be done extremely carefully for two reasons: 1) the risk is 'looking for the key under the streetlamp because it'd better light', and 2) unintended side effects.
The first will cause data to just be basically irrelevant or even inverse to the desired signal, but people will chase it anyway because it is easy to measure. The second taps into the management phenomena of "you get the results that you measure", and create perverse incentives to do the wrong thing, either self-managing, learning, or managing an org.
Good article, much more worthwhile than I expected!
I'm actually thinking that this is part of what is going on with all the cries of "Inflation!!" - compared to last years' numbers, when everything was running at about 40%, of course prices are rebounding, and even overcompensating on the high side - the real question is for how long (there was also a massive hole blown in the money supply, which needs to be backfilled).
[1] https://xkcd.com/882/
The novice bowler example has nothing to do the tightness of the feedback loop. The issue is solely the statistical variation in the results.
The novice bowler sometimes bowls too far to the left, sometimes to the right. This isn’t a result of conscious action, rather a lack of control. When the bowler aims for the middle of the lane and the ball randomly goes to the left on one occasion, aiming to the right on the next shot will not fix the problem; it will make it worse.
Summarizing the real issue for the bowler: don’t make decisions based on random data. If you don’t know to what extent the data you’re looking at contains a random element, you should try to figure out how much before attempting a correction.
Lengthening the feedback loop in this example might involve obscuring the bowler’s vision of the pins once the ball has begun rolling and presenting a video of the results 10 minutes later. It’s hard for me to imagine how this could be anything other than detrimental.
Back to the bowling example (sorry for harping on it), unless you frequently hallucinate and see the pins fall when they did not, the results in practice are very very accurate. It’s just that they describe a complex system with an initially unknown amount of variation. It’s only through having precise measurements that you can even quantify the variation.