>Suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people.
It was introduced to point out a place where the currently-best "decision theory" produces a seemingly-wrong recommended action in the hope that someone might come up with a better decision theory or alternatively to explain why the seemingly-wrong recommendation is actually right. A "decision theory" is a recipe for acting in the face of uncertainty that is precisely stated enough that it might eventually serve as the basis of a computer program.
This also touches mind theory. In particular it relies on the belief that running a Turing machine that kills 3^^^^3 people is as bad as breeding and killing the same number of people in the physical world we live in.
Pascal’s gamble wasn’t just about probability, it was about storytelling: the promise of nearly infinite payoff with minimal risk. That same allure is still at play today whenever people chase “moonshot” returns on crypto or quick-rich schemes.
It underscores a timeless lesson: no matter how much data or logic we have, we’re still wired to fall for well-crafted optimism and that means skepticism remains the best defense.
A lower-key variant of this frequently comes into play with consulting or other sales pitches. "You spend <big number> per year on this <necessary business expense>. Our service will easily shave 2% off this, making the cost of our service completely negligible and this purchase an obviously good decision."
This is exactly how I feel when Effective Altruism starts talking about the wellbeing of trillions of humans living in the far distant future that we should be devoting ourselves to now.
As with most cognitive biases, there's an inverse to this, where we ignore low-probability high-impact scenarios. E.g. people drive drunk or without a seatbelt, because it'll *probably* be fine. And they repeatedly have that assumption confirmed--until one day it isn't.
I had one friend who would leave his bike chained partially blocking a fire exit, because "what are the odds the fire inspector will come today?" But the fire inspector comes once a year, and if your bike is chained there 99% of the time, odds are you're going to get a fine. He couldn't see the logic. He got fined.
It's amusing to consider how much of a Rorschach test this article must be. But it's a great point, even if it arms us to abusively write off unwelcome ideas as scams. As the author points out, Pascal's reasoning is easily applied to an infinity of conceivable catastrophes - alien invasions, etc. That Pascal specifically applied his argument to the possibility of punishment by a biblical God was due to the psychological salience of that possibility in Pascal's culture - a truly balanced application of his fallacious reasoning would be completely paralyzing.
Rationalists and Effective Altruism people fall for this stuff _constantly_. Roko's Basilisk being the canonical example of it.
They assign infinite negative or postive values to outcomes and then it doesn't mighter what the likelihood or how much they uncertainty they have everywhere else, they insist that they need to do everything possible to cause or prevent whatever that outcome is.
Aside from other problems with it, there are a vast number of highly improbable and near-infinitely bad or good outcomes that might possibly occur which would require completey different actions if you're concerned about them.
As a kernel developer, I have always had to worry about what happens in the 0.000001% of cases. You're either correct all of the time or you likely have an exploitable security hole in your code that will turn into a CVE at some point in the future that will cost a lot of money to fix. One has to use strategies to reason through proof that your code does the right thing in all states, including the insertion of delays and other fault injection techniques to trigger the cases that aren't normally hit.
It's also kind of like saying that an LLM only pushes 0.01% of people off the deep end in a psychotic episode, so it doesn't really matter. That doesn't actually get rid of the liability.
All that said, I will always refuse to write life safety code. I am not a good enough developer to do so.
13 comments
[ 3.8 ms ] story [ 33.1 ms ] thread>Suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people.
It was introduced to point out a place where the currently-best "decision theory" produces a seemingly-wrong recommended action in the hope that someone might come up with a better decision theory or alternatively to explain why the seemingly-wrong recommendation is actually right. A "decision theory" is a recipe for acting in the face of uncertainty that is precisely stated enough that it might eventually serve as the basis of a computer program.
It underscores a timeless lesson: no matter how much data or logic we have, we’re still wired to fall for well-crafted optimism and that means skepticism remains the best defense.
https://www.effectivealtruism.org/articles/cause-profile-lon...
I had one friend who would leave his bike chained partially blocking a fire exit, because "what are the odds the fire inspector will come today?" But the fire inspector comes once a year, and if your bike is chained there 99% of the time, odds are you're going to get a fine. He couldn't see the logic. He got fined.
They assign infinite negative or postive values to outcomes and then it doesn't mighter what the likelihood or how much they uncertainty they have everywhere else, they insist that they need to do everything possible to cause or prevent whatever that outcome is.
Aside from other problems with it, there are a vast number of highly improbable and near-infinitely bad or good outcomes that might possibly occur which would require completey different actions if you're concerned about them.
It's also kind of like saying that an LLM only pushes 0.01% of people off the deep end in a psychotic episode, so it doesn't really matter. That doesn't actually get rid of the liability.
All that said, I will always refuse to write life safety code. I am not a good enough developer to do so.