That is, my understanding of reality is the same, regardless. I think the goals of "The Brights" are laudable, generally. But I didn't register on their website, and wouldn't really want my photo and identify featured as a Bright, because the name just doesn't feel right to me.
How you identify yourself makes a difference to how you're judged, and hence the influence you can bring to bear on the world.
Or, you know, if I someday gain a bit more public recognition and clout, it may start to matter then how I self-identified back in 2010.
That Eliezer Yudkowsky is a pretty smart guy. Against all prior expectations (lol), I am now reading Harry Potter fan fiction and enjoying it.[1] The LessWrong sequences are poignant and concise and pushed me down the road of rational thought.
I look forward to getting some of these books on my shelf.
This is a collection of Yudkowsky's blog posts on LessWrong, and it's making the rounds now because it forms the basis for most of Harry's rationalist insights in the extremely entertaining and recently finished Harry Potter and the Methods of Rationality. (The last chapter of HPMoR went up on Ultimate Pi Day - 3/14/15 - last week.) I picked it up on the advice of some friends, and while the editing is less than perfect, it is overall an impressively well-organized and cross-linked treatise on how to reason.
At the heart of the book is the assertion that rationality is not about justifying why we are right, but thinking how we might be wrong, and paying careful attention to any evidence that puzzles us or could convince us to change our minds. The resulting philosophy is a combination of the falsification principle and the scientific method, applied to your own character. Highly recommended.
Given that this compilation was published on March 11th, and hpmor finished on March 14th, I don't think it's making the rounds now because of the timing of hpmor. Their finish/release dates are simply close together.
I think it still may be - people are posting this around as an additional reading for those who liked HPMOR; number of people who read that fanfic is most likely much greater than the number of people that read the Sequences.
I quite liked his introduction! Be aware, however, that he dramatically overstates some things. For example, he claims that the many-worlds interpretation is obviously correct. This is basically wrong: it may be correct, but it certainly isn't obviously correct, and many scientists whose entire research is dedicated to quantum mechanics would disagree that many-worlds is correct at all.
There is a long list of alternative explanations[0], with very little evidence to lend support to or contradict any of them (at least, any of the ones that are still around. Some have fallen by the wayside).
I think it's fair to say that it's "obvious" when optimizing for the same things Yudkowsky does. For instance, any issue having something to say about the Born rule seems to be his dump stat, if you will. This makes sense, because in the paper where Max Born introduced the rule, he suggests that it indicates only one possible correct interpretation (hint: it was not Many Worlds).
That all being said, no one really disagrees about the predictions of the math, and at least many-worlds has less completely ridiculous misinterpretations by lay-people, so both ends of the spectrum are tolerable to me.
There is a much reduced and heavily edited sequence on quantum mechanics in the ebook. I have not yet read the ebook version and can't tell you how they differ, but the editor did state upfront that it was one of the sections which changed the most.
Regarding the LessWrong/OvercomingBias version, I must say that you should take the blog posts with a huge, gigantic, over-sized helping of salt. Speaking as someone who majored in physics because I love the field even though I knew it wouldn't help my comp sci career, Yudkowsky does not have the necessary background to speak with authority on this topic and it shows. He makes a few outright mistakes and a lot more overly broad statements, he ignores many validly competing alternative explanations, and as a result the grand philosophical conclusions reached are left unjustified (and as an editorial aside, probably bunk IMHO).
The important result of analyzing quantum mechanics as a case study in rationality is the valid point that if like most people (and sadly, most physics professors) you think quantum mechanics is weird, that should be very concerning. Quantum mechanics is real: it very accurately describes reality. If you have a problem with quantum mechanics, you have a problem with reality, and that's about as anti-rationalist as you can get. To improve, find out what part of you rejects the reality of quantum mechanics in spite of experimental evidence, and censor that.
Once you've achieved that step, you can skip the rest IMHO.
Last time I heard this particular bark, the bite was remarkably weak. [This Physics Overflow question](http://physics.stackexchange.com/questions/23785/what-errors...) didn't turn up any "outright mistakes" unless you count one very trivial sign error that doesn't change the thrust of any of the arguments.
There is an ebook here (http://lesswrong.com/lw/72m/an_epub_of_eliezers_blog_posts/) where you can read it unedited as it was originally written. I don't know if anyone would want to do this, but it's how I read them. I imagine a lot of the content would be edited down or out, especially the standalone stuff. That's not necessarily a bad thing though.
I have edited that post to recommend the book over my ePub. You can always read my compilation after finishing the book, if you want to read the articles the book leaves out.
Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk)
Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments.
> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe.
I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally never existed anywhere except in the imagination of David Gerard."[1]
"This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?
At least the part where it's implied that "Someone from organization X used the basilisk to convince innocent bystanders to donate to X" is false, for any value of X that I know of. Pascal's Wager is/was (I imagine?) used to get people to subscribe to various churches and give them money.
Well, yes? If Roko's basilisk was true, it'd be unethical to spread. It turned out to not be true, but if Roko was reasoning from the premise of it being true, or even possibly-true, then the action was still unethical.
More like, "he believed that he'd accidentally invented an eternal torture machine and then he tried to turn it on, and I yelled at him even though the eternal torture machine didn't work."
It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
If you think you've discovered a serious info-hazard and your first action is to post it publicly, you are not someone I want frequenting the same online spaces I frequent. Whether that particular info-hazard is, upon examination, remotely hazardous is aside from the point.
(If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
I believe there is a secret government agency with the purpose of controlling the entire world will come about.
I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities.
In light of this, it's important to keep knowledge of this future event from people so they aren't targeted.
Whoops, I just posted this to a public forum.
Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.
People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.)
EX: What's the maximum computational power of the universe?
Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real; that was why it was a big deal.
It's a concept coined by Nick Bostrom. Basically, some information can be harmful. As a simple, non-controvertial information hazard, imagine you learned an identity of a covert assasin of a foreign government, and you start telling about it to people around you. Don't be surprised if not only you drop dead, but the people you interact with as well.
You should familiarize yourself with that feeling so you can recognize it later; it's the feeling of falling into an inferential gap. (The thing might be worthless, true, like Scientology. I usually find that resolving confusion is worth my time, though.)
"Information hazard" is just a term for "things you or others learn and then are worse off because of". Examples of information hazards:
- Instructions on how to make a gigaton bomb with processes and materials anyone can acquire and assemble privately.
- A reachable-by-a-country mechanism for blowing up the sun.
- An image designed to temporarily or permanently mess with your vision. Sharing [1] on reddit as a "cool optical illusion" would be mean.
- Probably whatever a computer spits out if you give it a scan of your brain and set it on the task of finding a paragraph/image/movie/whatever that maximizes dopamine production.
- Some people would consider a devastating argument against free will to be an information hazard, predicting that people would then do whatever they wanted because they "couldn't be blamed".
- If you're a hardcore atheist: religions. If you're a hardcore theist who thinks the devil is out to trick you: convincing arguments against belief.
That's like saying you shouldn't be mad at someone who attempted to murder you because they failed. There's a reason we have a crime called "attempted murder". You may find the wikipedia article on "Inchoate offense" interesting, specifically the section on Impossibility[1].
If I learned that a co-worker believed that nuts were invariably fatal to me and gone on to hide some in my food, the nuts would do me no harm but I would still take the whole incident seriously.
The visible universe? Some back-of-the-napkin estimates are that no more than 2^500 bits will be flipped in the region of space we can see, but there's a crucial assumption about minimum temperature you need to make to make that that estimate finite.
On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?
What an absurd thing to consider. There is no possibility of building a computer the size of the universe. This website is full of crazy people talking about crazy things!
Also, there is this concept of "thought experiments". Building a universe-sized computer may sound as likely as sitting on the light wave and looking around, and yet the second one brought us Relativity.
The lesson I learned from Roko's Basilisk event that I think Eliezer and most of the LW community also endorses is that if you're not careful, applying some models too far can have those "extreme, nay dire consequences for certain thought experiments" - so it's best not to commit too far to those methods, because they may have some caveats, or you might just have applied them wrong.
Yet in this same thread here there is a discussion of torture and dust specks. As if the mystical power to prevent dust specks is close enough to reality to be interesting.
I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline.
(If you believe that the substitutes for coal power are available and superior, then taking it offline should still be a net good, one that is distributed over a great many people, with the downside (jail) concentrated on the saboteur.)
It's actually a fair question. But 1) I don't believe substitutes for coal power are immediately available, and taking out a plant would likely lead to many deaths (think: accidents caused by sudden blackout, rural hospitals without enough fuel for backup power generators) and inconvenience (think: areas with interrupted electrical service for months as people try to fix the grid and bring in additional power sources); 2) downsides of a coal plant aren't that bad in comparison.
Note that dustspeck vs. torture example involved 3^^^3 dustspecks, which is more than number of atoms in the universe raised to the power of number of atoms in the universe. It was purposefully chosen as such an extreme number, as to clearly outweight the utilitarian calculation in favour of torture - so that the example could be used to study our moral intuitions.
I always found it as the "how much my gut reaction matches with what the mathematical model suggests" kind of puzzle - useful for both exploring your own biases and the limitations of the model.
That's just what I mean, you end up examining your beliefs about what your gut reaction would be. Holding the knife in your hand and deciding what to do with it is when you find out your real gut reaction.
But I don't really know if we want to build a moral theory upon my gut reactions to things. My gut reactions are probably inconsistent. They are optimized by experience, culture, and evolution, not for things vastly outside the scope of common everyday situations.
I know I'm not a 100% selfless altruist. I don't donate all my money to charity and I'm not going to risk going to prison to sabotage a power plant. But that doesn't mean I can't aspire to being more altruistic or want the world to be more utilitarian.
Almost all of EY's posts are about how your gut reaction to things is often wrong, inconsistent, or leads to outcomes you probably don't desire. E.g. Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/) where people will pay more money to save 4,500 people out of 11,000, rather than 4,500 out of 250,000.
Right -- gut morality is strongly based on small, local interactions.
If someone invites you over for dinner and they have a live calf there, nursing at its mother, and they're intending to slaughter it in front of you, it's not the same (to gut morality) as if they slaughtered it the day before, and really not the same as if they just bought some veal at the supermarket (i.e.: paid someone out of sight to do it).
Likewise; buying something that was produced in a way that involved significant human misery and/or death feels awful to gut morality if the sweatshop, diamond mine, or whatever is next door. On the other side of the world? Eh... well this one is $6 cheaper...
Why should it be so different? Well, gut morality isn't logical, at all. It's worth a bit of effort to do better.
Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).
Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".
At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems an overly simplistic model to put it on a linear scale, when there seem to be negative experiences that we easily "bounce back from" and others that we don't.
When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture.
Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
Right, that's why I reject the assumption that you can multiply it like that.
Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?
The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison."
But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present).
And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
> What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?
I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.
But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.
Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.
Which do you choose? Sixty years from now, which do you wish you'd chosen?
> I wouldn't pay a penny to avoid the dust speck
Do you mean just your own dust speck? The question was all 3^^^3 of them.
If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
> Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero
Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.
One reason is that you'd need to prove that the lastic impact is in fact zero. 3^^^3 dust specks is enough for one of them to rub something in a wrong way that rubs something else in the wrong way that gives a person cancer.
Having a speck of dust in one's eye will change the course of life in a tiny way for any given person, true, but unless we're postulating an even more absurd edge case like "everyone blinking at exactly the same time makes a blip in the physics experiment machine go unnoticed and a week later the universe explodes", it's a change that's going to be effectively random and indistinguishable from the general minor messiness of human life.
You're not making a serious effort to guess at what difference it might make, though.
Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.
The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.
Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.
I think the dust speck argument would be better if first framed as a preference for yourself:
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
If you opt to remove the SPECKS, you would have to change some law of nature or some property of matter, which will have other consequences for the amount of suffering in this world. As you cannot reliably predict or calculate those consequences, you cannot solve this problem.
What, preference utilitarianism? Afaik, that does get discussed, especially in the Fun Theory sequence iirc. If you want some more discussion on torture vs. dust specks, I think nostalgebraist on tumblr had a discussion of it with slatestarscratchpad (again, iirc).
3^^^3 people isn't "a very large number." The population of Earth or a galaxy full of inhabited planets would be a very large number. 3^^^3 is an incomprehensibly large number which is why it can lead to odd outcomes. If it were merely something like everyone on Earth suffering a dust speck then of course you choose dust.
Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.
It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong.
Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.
> A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.http://edge.org/conversation/jaron_lanier-the-myth-of-ai
> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone.
I fail to see how the first part of this sentence implies the second.
That is indeed a common place where people disagree.
The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.
Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.
> The basic reason it should follow is that not doing so makes you inconsistent and exploitable
I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them.
- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
> EY's definition of good is ruthlessly utilitarian,
Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.)
With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?
The most common example I see is the dust-specks vs torture thing[1], where Eliezer thinks increasing dust specks eventually outweighs the torture and various other people think torture outweighs any number of dust specks. Eliezer is basing his position on taking a few decision theory axioms as obviously true, and others are basing their position on taking their moral intuitions as obviously true (i.e. a "One's modus ponens is another's modus tollens" situation).
Yea, I agree with your interpretation on the root cause of the disagreement. I don't mean to weigh in one way or the other, but I would like to point out that people's mathematical intuition when it comes to infinity or obscenely large numbers can be arbitrarily wrong. It takes a lot of training to even recognize the mistakes you're making, let alone correct for them.
I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing eternal torture for your forum readers? Who cares that the idea won't actually guarantee any such thing, Roko _thought_ that it might; I would be pissed as hell.
Anyway, my point is, I'm only offering a gentle and hopefully reasoned disagreement with someone I regard highly. I'm not jumping on the "fuck that guy" train. Moving on.
There was a post that boiled down to the question "Would you rather 3 ||| 3 people (where | is an ascii stand-in for Knuth's up-arrow notation) get a mote of dust in their eye, or one person be horrifically tortured for 50 years?" and his conclusion was basically "you can use math to assign some incredibly small epsilon of suffering to getting a mote of dust in your eye, but eventually, if you sum enough people, it's more suffering overall than one poor person getting horrifically tortured". My problem with his post was that I think that any morality scheme that results in the person getting tortured is fundamentally flawed, and I don't care how much math you throw at me to try and "prove" that it's better.
I don't think I've ever heard of an attempt to rigorously derive morality that I agree with. Morality is too contextual, too messy, for us to perfectly capture in those sorts of models. It's especially bad when we try and model morality mathematically, and then take as gospel the result of that model, rather than say "oh, um, that's not a great result, the model must be wrong".
That last point about the models actually captures a rather interesting point. Generally I agree that if the model isn't giving results that match observation than it should be concluded that the model is incorrect rather than the observation was flawed. This has worked very well as the scientific method, but I think that it is hinged on the fact that these observations are objective and verifiable.
Now I happen to agree with you that I feel this dust result is incorrect, but morality is so undefined and subjective that I can't help but disagree with your conclusion that the model must be wrong. The entire moral question of the dust vs torture doesn't have one defined answer so we can't really conclude anything more than opinions about the model
There aren't 3^^^3 atoms in the universe, let alone people. I think it's absurd to complain about a system not holding up when stretched far beyond the boundaries of the universe.
If you decrease the number to even the number of humans that ever existed/exist/will exist, then it returns to being negligible in relation to torture.
The problem with an if -> then, is when the 'if' is flawed, it doesn't really matter what the 'then' is. The human mind can't grasp 3^^^3 dust motes because it doesn't exist in any meaningful way.
>But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you.
>And similarly, if you had to choose between a googol people tortured for 49.9999999 years, and a googol-squared people being tortured for 49.9999998 years, you would pick the former.
>A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.
>If you find your preferences are circular here, that makes rather a mockery of moral grandstanding. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere. Maybe you think it a great display of virtue to choose for a googolplex people to get dust specks rather than one person being tortured. But if you would also trade a googolplex people getting one dust speck for a googolplex/googol people getting two dust specks et cetera, you sure aren't helping anyone. Circular preferences may work for feeling noble, but not for feeding the hungry or healing the sick.
He's assuming linearity, which at the very least needs justification. He's assuming that the function that maps from the pair (number of people, type of torture) to suffering is linear in the number of people, and also linear in the type of torture. I don't believe either of those things are true.
To put it more clearly, he says:
> So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort
And I don't agree that you can. The difference between dust-mote and torture is not one of degree, but one of _kind_. It's a discontinuous function (in my opinion). I don't know where the discontinuity is, but it's there.
There are a finite number of possible brain states, far less than the 3||3 number. All possible "feelings" is likewise less than that.
You need to be dividing all possible feelings of pain into two groups, and asserting that the absolute worst of the first group (containing dust specks) is incommensurate with the absolute best of the second group (containing torture).
At some point you need to say that you'd pick specks even over 1 minute of torture, even over 1 second, on which I think most people's intuitions would stop saying that. Or you need to find some point between 50 years and 1 second where it become commensurate.
Sure, they are commensurate at some point. I'd pick one nanosecond of torture over the dust-motes, for instance. But I'm not certain that I would ever choose the 50 years of torture over the dust motes for any number of dust-moted people. That's because it's not continuous, so you can't do the sort of epsilon-delta proofs that Yudkowsky's argument depends on.
That means that for some amount of time, say X seconds, you would prefer X-1 seconds of torture for one person over specks, but prefer specks over X seconds of torture for one person.
But let's double each side; presumably you would make the same decision if asked again, right? So now you prefer X-1 seconds of torture done to each of two people, over twice as much specks.
Now, unless X is very low, you should prefer X for a single person over X-1 for two people. (If you disagree with this, please give a plausible value for X that makes it false.)
So you prefer X on a single person over double!specks, but prefer single!specks over X. This seems extremely unlikely. Or even if true, we should be able to make you pick torture for 50 years just by multiplying specks another couple of orders of magnitude.
We're social animals. There's an amount of discomfort (maybe not large) I would be prepared to undergo to help someone else, and I find myself thinking that others "should" also be prepared to undergo such an amount of discomfort to assist others. Given the choice to accept a dust mote temporarily in my eye as part of a huge crowd in order to save another person from torture, I would gladly accept that, and I think all reasonable people would too, therefore there is no number of people such that I believe the utility of dustmote vs torture turns out in favour of torture.
You aren't quite grokking the difference between "huge crowd" and 3||3. How do you deal with the argument from circularity? Are your preferences circular, and if not, which part do you reject?
The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone.
The answer to the argument from circularity is obvious - there is a discontinuity. I can see the discontinuity in my own thinking, and you probably can too. I would accept a dust mote to save an individual from years of torture (and I think almost all reasonable people would), but I would not accept a dust mote to save two people from a dust mote. That may be unethical of me, since I would of course prefer the world that has fewer people with dust motes, but for me to expect the greater number of people to make the sacrifice, I must be prepared to make the sacrifice myself and it must be a sacrifice that I think all people should make. Exactly how much I think people should sacrifice for others is difficult to say, but there is a clear step change at some point.
I know that that is the formulation of the question, But since you're asking me to make the choice, you're asking me to inflict suffering on many extra people who would not have suffered otherwise. If what is gained by their suffering is large enough compared to that suffering that I believe they all (or nearly all) would have chosen voluntarily to accept the suffering for the benefit, then I am happy with the choice, and my best model for that is what suffering I would be prepared to undergo for what benefit. I think the confusion is between evaluating the utility between two possible alternatives (where obviously the world with fewer people suffering is better) vs evaluating the utility between two possible alternatives where those were the only two alternatives. If it's just two of many alternatives, the utility placed on a meaningful sacrifice doesn't come into it, but if it's a choice, the utility placed on a meaningful sacrifice does.
I don't see how that's relevant to answering the circularity argument. How are you avoiding the claim that your preferences are inconsistent? At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.
(Also, if you click on the comment you can reply).
There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else.
>At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.
Well, the boundary is fuzzy, so I believe that different people will draw the line in different places, but yes, that is correct. There is a point at which I accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.
I think the debatable range is actually quite large, so the tiny decrease part might not be fair (there are a lot of degrees of pain I would not demand someone to suffer to save others), but at that point of expected sacrifice, the number of people jumps to infinite.
> and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.
That seems like a good way to put it.
The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one speck in their eye to save a stranger from fifty years of torture".
So first of all, your preferences are still circular, unless you bite another bullet somewhere. But even your argument isn't quite accurate. It's not this one person who needs to get a speck, it's a literally unimaginable amount of people.
As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
> But even your argument isn't quite accurate. It's not this one person who needs to get a speck, it's a literally unimaginable amount of people.
Nearly all of whom prefer to receive the speck than to allow the individual to be tortured.
> As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
Yes, I feel somewhat uncomfortable ignoring the agency of torturers and murderers in this scenario, and that reluctance would play into a very conservative estimate of where that threshold should be, but I would be reluctant to choose a lowest common denominator measure to establish morality.
So your intuition says that there's some point where everyone is suffering pain X, where X is very large, and they would each agree to cause the X-(1 mote speck) to a trillion times as many people, in return for reducing their own suffering by 1 mote speck? That strikes me as beyond regular selfishness, and non intuitive.
Can you explain more clearly why what I said implies that, because I don't think I mean to say that.
I believe that there is an amount of suffering that it is reasonable to expect anyone to accept in order to help another person. That amount depends on the amount to be suffered, and the amount benefited by the recipient. Once the suffering falls under that threshold, I do not believe the number of people required to make the sacrifice comes into consideration, as each of them if reasonable would say "I prefer to belong to this world, where as part of a huge group I accept this small ill in order that someone else benefits". Therefore, the implied sacrifice results in greater utility for that choice.
Let me try a different tack.
Let's say that you observe a universe with some large number of people suffering dust specks in their eye. That sounds bad. But what if every single one of those people actually suffering thinks that this universe is better than the alternatives. You don't suffer from a dust spec, but are you going to ignore all those people in their estimation of the utility of the universe? If you switched to a universe where all of those people didn't suffer from dust specs, but someone else was suffering, they would tell you that that was a worse universe.
It's pretty obvious to me that even if that isn't the exact case, it's close to being the case in reality - that's why people find the dust speck argument to be unintuitive, not the large numbers thing. It's because some measure of sacrifice for other people is part of what we expect from everyone, and most people know instinctively that if everyone asked to make a sacrifice agrees that it's right to make that sacrifice, then the world is better because of it.
This is not really a good format for continuing this discussion, because lack of notifications and such. Would you consider opening an account on lesswrong and posting in open thread? Or you could PM my account there at http://lesswrong.com/user/ike/overview/. That said, here's my reply.
>Can you explain more clearly why what I said implies that, because I don't think I mean to say that.
It's basically a reformulation of the circularity argument.
I assume there's some level of pain that you would prefer to the specks; say a single second of torture, equivalent to a smack or such. (If you think we should prefer 3|||3 specks to one smack, I could go further, so let me know.)
So counting up from that one second at a time (i.e. 2 seconds of pain, 3 second, etc), eventually we reach a point where you no longer think it's better than the specks. Call this X.
So X and X-epsilon are qualitatively different; the lower amount is not bad enough to outweigh specks, but the higher amount is. You need to prefer giving X-epsilon to a large number of people rather than X to a single one, if the qualitative difference is to be upheld.
(I may not be phrasing this so well. Maybe try working through the circularity argument above, or the other phrasings I used in this thread.)
Now to respond to your line of reasoning: this proves too much.
Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture? Have you donated all the money you could get to prevent torture? (and yes, I'm sure there are charities that are at least partially effective.)
Why doesn't your argument work for the case I just outlined as well?
> I assume there's some level of pain that you would prefer to the specks; say a single second of torture, equivalent to a smack or such.
Quite possibly I prefer a world with one person tortured for a lifetime compared to 3|||3 specks if they are evaluated out of context. It's hard to say, because pain doesn't easily sum, there are different qualities of pain, and we are talking about situations where we may be losing an entire persons contribution to humanity. I just don't think any of this is relevant to the conversation, or to the reason that so many people find your conclusion unpalatable - and it's nothing to do with not understanding large numbers.
In the case that they are evaluated in the context of a choice between one of those worlds or the other world, I would take into account what I believe to be the value that the individuals involved would place on the worlds were they to know the details and have minimal moral standards like mine.
Let me phrase it another way:
What would you say the utility of a world where there are 3|||3 people with specks who chose it gladly and voluntarily to save someone from torture is?
Let's say you tell those 3|||3 who wanted to save someone from torture by accepting a speck in their eye that they cannot, and someone must be tortured instead. You've massively increased the unhappiness in the world - not only is an individual getting tortured, but 3|||3 have ended up with a situation that's worse than they wanted. Are you going to claim that it's still got a higher utility? Now that you notice that you're making those 3|||3 unhappy by the choice, you can see that the disutility scales with the number of people - that's why the number of people becomes irrelevant.
> Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture?
Your phrasing is unnecessarily emotive here. You seem to be saying that 3|||3 dollars is an awful lot of dollars without giving me any context about what fraction of the dollars belonging to those 3|||3 people those 3|||3 dollars are or what else they could/should be spending it on. If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes. You'll notice that when it's phrased like that it does not require that I donate all the money I could get to prevent torture. I in fact do donate a small amount of money regularly to prevent torture, but I would demand much less of the 3|||3.
>Let's say you tell those 3|||3 who wanted to save someone from torture by accepting a speck in their eye that they cannot, and someone must be tortured instead. You've massively increased the unhappiness in the world - not only is an individual getting tortured, but 3|||3 have ended up with a situation that's worse than they wanted. Are you going to claim that it's still got a higher utility? Now that you notice that you're making those 3|||3 unhappy by the choice, you can see that the disutility scales with the number of people - that's why the number of people becomes irrelevant
If the people are told of the choice, that's a whole new problem, but that's kind of avoiding the point of the original question. To use a hacking analogy, you're using a side-channel to cheat.
Nobody is told about any of this. If they were, that would itself need to be factored in, and quite possibly lead me to prefer specks.
>If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes.
Each person isn't donating to save someone from torture, they're donating to save 1/3|||3 of torture.
Let's rephrase the original question to zero in on that last point. There are 3|||3 people. You choose the number of people who donate one dollar, which can be any number between 0 and 3|||3. After you make a decision, one of those people is chosen at random, and they are tortured iff they did not donate.
If you think about it, this results in the exact same outcomes in either choice, except you have more options than all or nothing. You basically choose the probability of torture.
To be consistent with your previous view, you'd need to not pick zero donaters. So for at least some people, it should be worth it for them to pay 1 dollar to avoid a 1/3|||3 probability of torture.
This dissolves, as you may have noticed, into Pascal's Wager. (Or Pascal's Mugging, more precisely, which was coined by Yudkowsky.) What if I tell you that unless you give me a dollar, I'm going to torture you for 50 years? The probability of that being true is more than 1/3|||3 (and if you disagree, then you are way too overconfident for life. 3|||3 is a huge number, and there's no conclusions I can think of in which I'd place that much confidence. We don't even have anywhere near the kind of raw data to draw any conclusion that confident.) So are you willing to give up a dollar to avoid the >1/3|||3 chance that I'm telling the truth?
If not, and all or most of those people in the problem would answer the same, then your previous rationale falls apart.
(Oh, and this is not the real Pascal's Mugging; that's much harder to deal with. But let's stick with the easy stuff for now, shall we?)
> Nobody is told about any of this. If they were, that would itself need to be factored in, and quite possibly lead me to prefer specks.
Exactly. My whole contention is that the reason this question is considered unintuitive by so many people is that they're really considering a different question to the one you think you're asking.
> I would gladly accept that, and I think all reasonable people would too
Hello, apparently I'm unreasonable. And so is everyone else who has said that they choose torture over dust specks. If you select 3^^^3 people, you're going to find an awful lot of us. (And also some sociopaths who literally don't care if someone else gets tortured.)
Your argument seems to boil down to "specks is the correct answer, so anyone who gets it wrong doesn't count; and because we all agree that specks is the correct answer, it's okay to do specks".
On the other hand, I would totally accept the torture for myself, if it would prevent the specks. (At least I hope I would, and to the extent that I can model how I would act in that situation, it does seem plausible that I would.)
But that's assuming that unpleasantness per individual is a continuous function always distinguishable from normal human experience, rather than something like a line eventually rising from the discontinuous murky soup that contains all the constant minor annoyances of being human (itchy nose, wedgies, the feeling of the back of your tongue against the roof of your mouth, one shoe being a little tighter than the other, etc) that the brain is well-optimized to tune out and rapidly forget about after the fact.
Even if it's discontinuous, it still needs to grow incredibly slowly. The ratio of pain neurons that fire for a speck versus for torture over 50 years is nowhere near 3||3. You need a function that grows so slow that outweighs 3||3, which is damn near impossible for any plausible function.
I care about impact on people, not neuronal activity, and the point I'm making is that there's a minimum below which the real impact on a given person, considered in aggregate with the impact of everything else that's a part of being human, is effectively nil.
Actually my decision probably depends on the person. [cough]
But anyway. This isn't even philosophy - it's a digital remix of medieval scholasticism pretending to be philosophy.
The irony is that politics proves empirically that ideas actually can be dangerous and harmful. And some ideas - actually narratives - can be very dangerous and harmful indeed.
There's over a century of "persuasion technology" (Bernays, etc) that exploits this.
Nothing I've seen Y write deals with the problem of politics as a social exploit in an insightful - never mind a useful - way.
Meanwhile real people are being tortured in real ways. What's his proposed rational solution to that problem?
I notice you didn't actually say at which point you would prefer torturing (10^100)*X people for Y years each, over torturing X people for Y+0.0000001 years each, for X and Y at least 0.0000001.
(You may assume you don't know anything in particular about these people, other than that they are adult humans.)
What the heck? Sorry, I didn't understand any of this. Are you saying that with real moral issues, it always trivially wrong to torture? This seems simply false.
If I capture person X and X's laptop Y, and X tells me under no duress that Y contains the location of a nuclear bomb that X has placed in a major city, and I have other strong evidence that this is true, but X refuses to give me the password to Y; then it is moral (but rightly illegal) for me to torture X for the password to Y.
Torture is not literally incommensurate with any other bad thing. Then the question arises, how do we, in full generality, determine which is the greater of two evils (or the better of two goods)? The torture vs. dust specks thing is supposed to disabuse people of the unhelpful notion that some things are somehow incomparable in terms of goodness and badness.
I think the dust speck argument would be better if first framed as a preference for yourself:
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
Yes, I agree, I would take the risk, but those are two entirely different things. It's not the same argument framed in a different way at all.
In Yudkowsky's argument, it's the option between one person _absolutely guaranteed_ to have 50 years of torture, vs 3^^^3 people _absolutely guaranteed_ to be dust-moted. I think the guarantee changes things significantly.
If you consistently take low risks that might result in getting tortured, then you are pretty much guaranteed to eventually lose the bet and suffer. The probability of eventually losing even a small bet approaches 1 if you take it enough times.
Just think of it as if you were going to other people and deciding to take this bet for them. 1/3^^^3 chance of torture in exchange for removing a speck of dust from their eye. I think you would be ok with that because it's an obvious choice for ourselves.
And if you continue to do this for enough people, eventually one of them will get tortured. After removing dust specks from approximately 3^^^3 people's eye. But you are right, there is a chance no one will get tortured. So do it for 3^^^^3 people then and the probability is 0.99999...
If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous).
Something as innocuous as "Think safe thoughts!" enjoys a similar property.
That doesn't make sense, though -- info hazards are not normally things you might just think of, or we'd be screwed either way; "whatever you do, don't think of a pink elephant", right?
Toss out silly ideas like the basilisk thing, and think up real info hazard possibilities.
E.g., let's say you knew what Snowden was up to, a few weeks before he figured out how to publish what he'd found and leave the US.
You'd have done a lot of harm to post that info publicly.
Suppose you find a laptop left in a taxi, with police records including personal details of rape victims in your city for the past 20 years. Post it to 4chan? Uh, no.
What about ongoing undercover investigations? Post it?
You accidentally shoulder-surf sensitive credentials -- bank manager's login, nuclear plant access code, whatever. Post them online?
Simple ways to make highly dangerous weapons, drugs, poisons, etc. can also be info hazards.
It's honestly not that hard to imagine real-life examples, without having the dangerous info in question or descending into government assassin-type scenarios.
It's just any data that, if you make it generally available, will predictably result in bad things happening.
In some cases -- like revealing important security flaws -- a public disclosure may be a last resort if less harmful methods have failed, but it's likely not an ethical first resort in those cases.
I'm happy to see this in the preface (where EY looks back and details what he thinks he did wrong):
"Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt.
Despite my mistake, I am happy to say that my readership has so far been amazingly good about not using my rhetoric as an excuse to bully or belittle others. (I want to single out Scott Alexander in particular here, who is a nicer person than I am and an increasingly amazing writer on these topics, and may deserve a part of the credit for making the culture of Less Wrong a healthy one.)"
gwern, Scott makes a specific point of not linking to his blog from his LJ in his penultimate LJ post - he might prefer that you didn’t make that link here?
141 comments
[ 3.5 ms ] story [ 199 ms ] threadVague counterpoint -- this is an ebook collecting essays from lesswrong.com, which is a bit more subtle as names go.
A name is enough of a reason on it's own?
That is, my understanding of reality is the same, regardless. I think the goals of "The Brights" are laudable, generally. But I didn't register on their website, and wouldn't really want my photo and identify featured as a Bright, because the name just doesn't feel right to me.
How you identify yourself makes a difference to how you're judged, and hence the influence you can bring to bear on the world.
Or, you know, if I someday gain a bit more public recognition and clout, it may start to matter then how I self-identified back in 2010.
I look forward to getting some of these books on my shelf.
[1] http://hpmor.com/
At the heart of the book is the assertion that rationality is not about justifying why we are right, but thinking how we might be wrong, and paying careful attention to any evidence that puzzles us or could convince us to change our minds. The resulting philosophy is a combination of the falsification principle and the scientific method, applied to your own character. Highly recommended.
Does anyone know whether that sequence has made it to one of these books?
There is a long list of alternative explanations[0], with very little evidence to lend support to or contradict any of them (at least, any of the ones that are still around. Some have fallen by the wayside).
[0]: https://en.wikipedia.org/wiki/Interpretations_of_quantum_mec...
That all being said, no one really disagrees about the predictions of the math, and at least many-worlds has less completely ridiculous misinterpretations by lay-people, so both ends of the spectrum are tolerable to me.
Regarding the LessWrong/OvercomingBias version, I must say that you should take the blog posts with a huge, gigantic, over-sized helping of salt. Speaking as someone who majored in physics because I love the field even though I knew it wouldn't help my comp sci career, Yudkowsky does not have the necessary background to speak with authority on this topic and it shows. He makes a few outright mistakes and a lot more overly broad statements, he ignores many validly competing alternative explanations, and as a result the grand philosophical conclusions reached are left unjustified (and as an editorial aside, probably bunk IMHO).
The important result of analyzing quantum mechanics as a case study in rationality is the valid point that if like most people (and sadly, most physics professors) you think quantum mechanics is weird, that should be very concerning. Quantum mechanics is real: it very accurately describes reality. If you have a problem with quantum mechanics, you have a problem with reality, and that's about as anti-rationalist as you can get. To improve, find out what part of you rejects the reality of quantum mechanics in spite of experimental evidence, and censor that.
Once you've achieved that step, you can skip the rest IMHO.
Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments.
Consume this philosophy at your own risk.
And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.
I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
[1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
See "Belief in belief", which is right near the beginning of the first sequence....
http://lesswrong.com/lw/i4/belief_in_belief/
It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
(If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
I believe there is a secret government agency with the purpose of controlling the entire world will come about.
I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities.
In light of this, it's important to keep knowledge of this future event from people so they aren't targeted.
Whoops, I just posted this to a public forum.
Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.
EX: What's the maximum computational power of the universe?
I feel like I just stumbled into a Scientology auditing session or something reading your comment.
http://www.nickbostrom.com/information-hazards.pdf
http://wiki.lesswrong.com/wiki/Information_hazard
- Instructions on how to make a gigaton bomb with processes and materials anyone can acquire and assemble privately.
- A reachable-by-a-country mechanism for blowing up the sun.
- An image designed to temporarily or permanently mess with your vision. Sharing [1] on reddit as a "cool optical illusion" would be mean.
- Probably whatever a computer spits out if you give it a scan of your brain and set it on the task of finding a paragraph/image/movie/whatever that maximizes dopamine production.
- Some people would consider a devastating argument against free will to be an information hazard, predicting that people would then do whatever they wanted because they "couldn't be blamed".
- If you're a hardcore atheist: religions. If you're a hardcore theist who thinks the devil is out to trick you: convincing arguments against belief.
1: http://en.wikipedia.org/wiki/McCollough_effect
But... for "might be real" to be valid at all, you have to buy into the Roko's basilisk concept in the first place.
If you don't, there's nothing to get upset about.
1: http://en.wikipedia.org/wiki/Inchoate_offense#Impossibility
On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?
Also, there is this concept of "thought experiments". Building a universe-sized computer may sound as likely as sitting on the light wave and looking around, and yet the second one brought us Relativity.
I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline.
(If you believe that the substitutes for coal power are available and superior, then taking it offline should still be a net good, one that is distributed over a great many people, with the downside (jail) concentrated on the saboteur.)
Note that dustspeck vs. torture example involved 3^^^3 dustspecks, which is more than number of atoms in the universe raised to the power of number of atoms in the universe. It was purposefully chosen as such an extreme number, as to clearly outweight the utilitarian calculation in favour of torture - so that the example could be used to study our moral intuitions.
Short of that, it's just a mind puzzle, "how does what I think my beliefs are match up with what I think my beliefs are" or so.
I know I'm not a 100% selfless altruist. I don't donate all my money to charity and I'm not going to risk going to prison to sabotage a power plant. But that doesn't mean I can't aspire to being more altruistic or want the world to be more utilitarian.
Almost all of EY's posts are about how your gut reaction to things is often wrong, inconsistent, or leads to outcomes you probably don't desire. E.g. Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/) where people will pay more money to save 4,500 people out of 11,000, rather than 4,500 out of 250,000.
If someone invites you over for dinner and they have a live calf there, nursing at its mother, and they're intending to slaughter it in front of you, it's not the same (to gut morality) as if they slaughtered it the day before, and really not the same as if they just bought some veal at the supermarket (i.e.: paid someone out of sight to do it).
Likewise; buying something that was produced in a way that involved significant human misery and/or death feels awful to gut morality if the sweatshop, diamond mine, or whatever is next door. On the other side of the world? Eh... well this one is $6 cheaper...
Why should it be so different? Well, gut morality isn't logical, at all. It's worth a bit of effort to do better.
At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?
The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison."
But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present).
And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.
But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.
Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.
Which do you choose? Sixty years from now, which do you wish you'd chosen?
> I wouldn't pay a penny to avoid the dust speck
Do you mean just your own dust speck? The question was all 3^^^3 of them.
If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
a) would you press the 7bn specks button for an individual 'reward' of $70million?
b) would you press the N-specks button if given a choice between that and the 1-torture button?
Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.
Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.
The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.
Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.
I think the dust speck argument would be better if first framed as a preference for yourself:
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.
Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.
> A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai
I fail to see how the first part of this sentence implies the second.
The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.
Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.
1: http://en.wikipedia.org/wiki/Allais_paradox
I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
That doesn't seem to be Yudkowsky's argument.
Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.)
With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?
The most common example I see is the dust-specks vs torture thing[1], where Eliezer thinks increasing dust specks eventually outweighs the torture and various other people think torture outweighs any number of dust specks. Eliezer is basing his position on taking a few decision theory axioms as obviously true, and others are basing their position on taking their moral intuitions as obviously true (i.e. a "One's modus ponens is another's modus tollens" situation).
1: http://lesswrong.com/lw/kn/torture_vs_dust_specks/
Anyway, my point is, I'm only offering a gentle and hopefully reasoned disagreement with someone I regard highly. I'm not jumping on the "fuck that guy" train. Moving on.
There was a post that boiled down to the question "Would you rather 3 ||| 3 people (where | is an ascii stand-in for Knuth's up-arrow notation) get a mote of dust in their eye, or one person be horrifically tortured for 50 years?" and his conclusion was basically "you can use math to assign some incredibly small epsilon of suffering to getting a mote of dust in your eye, but eventually, if you sum enough people, it's more suffering overall than one poor person getting horrifically tortured". My problem with his post was that I think that any morality scheme that results in the person getting tortured is fundamentally flawed, and I don't care how much math you throw at me to try and "prove" that it's better.
I don't think I've ever heard of an attempt to rigorously derive morality that I agree with. Morality is too contextual, too messy, for us to perfectly capture in those sorts of models. It's especially bad when we try and model morality mathematically, and then take as gospel the result of that model, rather than say "oh, um, that's not a great result, the model must be wrong".
Now I happen to agree with you that I feel this dust result is incorrect, but morality is so undefined and subjective that I can't help but disagree with your conclusion that the model must be wrong. The entire moral question of the dust vs torture doesn't have one defined answer so we can't really conclude anything more than opinions about the model
If you decrease the number to even the number of humans that ever existed/exist/will exist, then it returns to being negligible in relation to torture.
The problem with an if -> then, is when the 'if' is flawed, it doesn't really matter what the 'then' is. The human mind can't grasp 3^^^3 dust motes because it doesn't exist in any meaningful way.
>But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you.
>And similarly, if you had to choose between a googol people tortured for 49.9999999 years, and a googol-squared people being tortured for 49.9999998 years, you would pick the former.
>A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.
>If you find your preferences are circular here, that makes rather a mockery of moral grandstanding. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere. Maybe you think it a great display of virtue to choose for a googolplex people to get dust specks rather than one person being tortured. But if you would also trade a googolplex people getting one dust speck for a googolplex/googol people getting two dust specks et cetera, you sure aren't helping anyone. Circular preferences may work for feeling noble, but not for feeding the hungry or healing the sick.
To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort
And I don't agree that you can. The difference between dust-mote and torture is not one of degree, but one of _kind_. It's a discontinuous function (in my opinion). I don't know where the discontinuity is, but it's there.
There are a finite number of possible brain states, far less than the 3||3 number. All possible "feelings" is likewise less than that.
You need to be dividing all possible feelings of pain into two groups, and asserting that the absolute worst of the first group (containing dust specks) is incommensurate with the absolute best of the second group (containing torture).
At some point you need to say that you'd pick specks even over 1 minute of torture, even over 1 second, on which I think most people's intuitions would stop saying that. Or you need to find some point between 50 years and 1 second where it become commensurate.
At least, in my opinion.
But let's double each side; presumably you would make the same decision if asked again, right? So now you prefer X-1 seconds of torture done to each of two people, over twice as much specks.
Now, unless X is very low, you should prefer X for a single person over X-1 for two people. (If you disagree with this, please give a plausible value for X that makes it false.)
So you prefer X on a single person over double!specks, but prefer single!specks over X. This seems extremely unlikely. Or even if true, we should be able to make you pick torture for 50 years just by multiplying specks another couple of orders of magnitude.
Does this make his argument any clearer?
The answer to the argument from circularity is obvious - there is a discontinuity. I can see the discontinuity in my own thinking, and you probably can too. I would accept a dust mote to save an individual from years of torture (and I think almost all reasonable people would), but I would not accept a dust mote to save two people from a dust mote. That may be unethical of me, since I would of course prefer the world that has fewer people with dust motes, but for me to expect the greater number of people to make the sacrifice, I must be prepared to make the sacrifice myself and it must be a sacrifice that I think all people should make. Exactly how much I think people should sacrifice for others is difficult to say, but there is a clear step change at some point.
So saying that you are selfish doesn't get you out of this, as long as you do prefer the world with less other people having specks.
Where is there a discontinuity when talking about other people being tortured or specked?
(Also, if you click on the comment you can reply).
>At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.
Well, the boundary is fuzzy, so I believe that different people will draw the line in different places, but yes, that is correct. There is a point at which I accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.
I think the debatable range is actually quite large, so the tiny decrease part might not be fair (there are a lot of degrees of pain I would not demand someone to suffer to save others), but at that point of expected sacrifice, the number of people jumps to infinite.
That seems like a good way to put it.
The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one speck in their eye to save a stranger from fifty years of torture".
As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
Nearly all of whom prefer to receive the speck than to allow the individual to be tortured.
> As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.
Yes, I feel somewhat uncomfortable ignoring the agency of torturers and murderers in this scenario, and that reluctance would play into a very conservative estimate of where that threshold should be, but I would be reluctant to choose a lowest common denominator measure to establish morality.
I believe that there is an amount of suffering that it is reasonable to expect anyone to accept in order to help another person. That amount depends on the amount to be suffered, and the amount benefited by the recipient. Once the suffering falls under that threshold, I do not believe the number of people required to make the sacrifice comes into consideration, as each of them if reasonable would say "I prefer to belong to this world, where as part of a huge group I accept this small ill in order that someone else benefits". Therefore, the implied sacrifice results in greater utility for that choice.
Let me try a different tack.
Let's say that you observe a universe with some large number of people suffering dust specks in their eye. That sounds bad. But what if every single one of those people actually suffering thinks that this universe is better than the alternatives. You don't suffer from a dust spec, but are you going to ignore all those people in their estimation of the utility of the universe? If you switched to a universe where all of those people didn't suffer from dust specs, but someone else was suffering, they would tell you that that was a worse universe.
It's pretty obvious to me that even if that isn't the exact case, it's close to being the case in reality - that's why people find the dust speck argument to be unintuitive, not the large numbers thing. It's because some measure of sacrifice for other people is part of what we expect from everyone, and most people know instinctively that if everyone asked to make a sacrifice agrees that it's right to make that sacrifice, then the world is better because of it.
>Can you explain more clearly why what I said implies that, because I don't think I mean to say that.
It's basically a reformulation of the circularity argument.
I assume there's some level of pain that you would prefer to the specks; say a single second of torture, equivalent to a smack or such. (If you think we should prefer 3|||3 specks to one smack, I could go further, so let me know.)
So counting up from that one second at a time (i.e. 2 seconds of pain, 3 second, etc), eventually we reach a point where you no longer think it's better than the specks. Call this X.
So X and X-epsilon are qualitatively different; the lower amount is not bad enough to outweigh specks, but the higher amount is. You need to prefer giving X-epsilon to a large number of people rather than X to a single one, if the qualitative difference is to be upheld.
(I may not be phrasing this so well. Maybe try working through the circularity argument above, or the other phrasings I used in this thread.)
Now to respond to your line of reasoning: this proves too much.
Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture? Have you donated all the money you could get to prevent torture? (and yes, I'm sure there are charities that are at least partially effective.)
Why doesn't your argument work for the case I just outlined as well?
Quite possibly I prefer a world with one person tortured for a lifetime compared to 3|||3 specks if they are evaluated out of context. It's hard to say, because pain doesn't easily sum, there are different qualities of pain, and we are talking about situations where we may be losing an entire persons contribution to humanity. I just don't think any of this is relevant to the conversation, or to the reason that so many people find your conclusion unpalatable - and it's nothing to do with not understanding large numbers.
In the case that they are evaluated in the context of a choice between one of those worlds or the other world, I would take into account what I believe to be the value that the individuals involved would place on the worlds were they to know the details and have minimal moral standards like mine.
Let me phrase it another way:
What would you say the utility of a world where there are 3|||3 people with specks who chose it gladly and voluntarily to save someone from torture is?
Let's say you tell those 3|||3 who wanted to save someone from torture by accepting a speck in their eye that they cannot, and someone must be tortured instead. You've massively increased the unhappiness in the world - not only is an individual getting tortured, but 3|||3 have ended up with a situation that's worse than they wanted. Are you going to claim that it's still got a higher utility? Now that you notice that you're making those 3|||3 unhappy by the choice, you can see that the disutility scales with the number of people - that's why the number of people becomes irrelevant.
> Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture?
Your phrasing is unnecessarily emotive here. You seem to be saying that 3|||3 dollars is an awful lot of dollars without giving me any context about what fraction of the dollars belonging to those 3|||3 people those 3|||3 dollars are or what else they could/should be spending it on. If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes. You'll notice that when it's phrased like that it does not require that I donate all the money I could get to prevent torture. I in fact do donate a small amount of money regularly to prevent torture, but I would demand much less of the 3|||3.
If the people are told of the choice, that's a whole new problem, but that's kind of avoiding the point of the original question. To use a hacking analogy, you're using a side-channel to cheat.
Nobody is told about any of this. If they were, that would itself need to be factored in, and quite possibly lead me to prefer specks.
>If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes.
Each person isn't donating to save someone from torture, they're donating to save 1/3|||3 of torture.
Let's rephrase the original question to zero in on that last point. There are 3|||3 people. You choose the number of people who donate one dollar, which can be any number between 0 and 3|||3. After you make a decision, one of those people is chosen at random, and they are tortured iff they did not donate.
If you think about it, this results in the exact same outcomes in either choice, except you have more options than all or nothing. You basically choose the probability of torture.
To be consistent with your previous view, you'd need to not pick zero donaters. So for at least some people, it should be worth it for them to pay 1 dollar to avoid a 1/3|||3 probability of torture.
This dissolves, as you may have noticed, into Pascal's Wager. (Or Pascal's Mugging, more precisely, which was coined by Yudkowsky.) What if I tell you that unless you give me a dollar, I'm going to torture you for 50 years? The probability of that being true is more than 1/3|||3 (and if you disagree, then you are way too overconfident for life. 3|||3 is a huge number, and there's no conclusions I can think of in which I'd place that much confidence. We don't even have anywhere near the kind of raw data to draw any conclusion that confident.) So are you willing to give up a dollar to avoid the >1/3|||3 chance that I'm telling the truth?
If not, and all or most of those people in the problem would answer the same, then your previous rationale falls apart.
(Oh, and this is not the real Pascal's Mugging; that's much harder to deal with. But let's stick with the easy stuff for now, shall we?)
Also, what about the argument in http://lesswrong.com/lw/kn/torture_vs_dust_specks/ueo
(And if we're going to continue this, could you please move it over to another forum? It's getting harder to keep track of comments here.)
Exactly. My whole contention is that the reason this question is considered unintuitive by so many people is that they're really considering a different question to the one you think you're asking.
Hello, apparently I'm unreasonable. And so is everyone else who has said that they choose torture over dust specks. If you select 3^^^3 people, you're going to find an awful lot of us. (And also some sociopaths who literally don't care if someone else gets tortured.)
Your argument seems to boil down to "specks is the correct answer, so anyone who gets it wrong doesn't count; and because we all agree that specks is the correct answer, it's okay to do specks".
On the other hand, I would totally accept the torture for myself, if it would prevent the specks. (At least I hope I would, and to the extent that I can model how I would act in that situation, it does seem plausible that I would.)
But anyway. This isn't even philosophy - it's a digital remix of medieval scholasticism pretending to be philosophy.
The irony is that politics proves empirically that ideas actually can be dangerous and harmful. And some ideas - actually narratives - can be very dangerous and harmful indeed.
There's over a century of "persuasion technology" (Bernays, etc) that exploits this.
Nothing I've seen Y write deals with the problem of politics as a social exploit in an insightful - never mind a useful - way.
Meanwhile real people are being tortured in real ways. What's his proposed rational solution to that problem?
(You may assume you don't know anything in particular about these people, other than that they are adult humans.)
The fact that it includes some numbers that reduce to some other numbers doesn't change that.
Putting numbers into something doesn't make it scientific or objective. It just makes it numerical.
There is a difference, and it's not a small one.
If I capture person X and X's laptop Y, and X tells me under no duress that Y contains the location of a nuclear bomb that X has placed in a major city, and I have other strong evidence that this is true, but X refuses to give me the password to Y; then it is moral (but rightly illegal) for me to torture X for the password to Y.
Torture is not literally incommensurate with any other bad thing. Then the question arises, how do we, in full generality, determine which is the greater of two evils (or the better of two goods)? The torture vs. dust specks thing is supposed to disabuse people of the unhelpful notion that some things are somehow incomparable in terms of goodness and badness.
If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.
I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).
I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.
There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.
And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.
In Yudkowsky's argument, it's the option between one person _absolutely guaranteed_ to have 50 years of torture, vs 3^^^3 people _absolutely guaranteed_ to be dust-moted. I think the guarantee changes things significantly.
Just think of it as if you were going to other people and deciding to take this bet for them. 1/3^^^3 chance of torture in exchange for removing a speck of dust from their eye. I think you would be ok with that because it's an obvious choice for ourselves.
And if you continue to do this for enough people, eventually one of them will get tortured. After removing dust specks from approximately 3^^^3 people's eye. But you are right, there is a chance no one will get tortured. So do it for 3^^^^3 people then and the probability is 0.99999...
Something as innocuous as "Think safe thoughts!" enjoys a similar property.
If I modded this, I would delete your post.
If I don't take info hazards as a serious threat, there isn't anything to sort those statements with.
Toss out silly ideas like the basilisk thing, and think up real info hazard possibilities.
E.g., let's say you knew what Snowden was up to, a few weeks before he figured out how to publish what he'd found and leave the US.
You'd have done a lot of harm to post that info publicly.
Suppose you find a laptop left in a taxi, with police records including personal details of rape victims in your city for the past 20 years. Post it to 4chan? Uh, no.
What about ongoing undercover investigations? Post it?
You accidentally shoulder-surf sensitive credentials -- bank manager's login, nuclear plant access code, whatever. Post them online?
Simple ways to make highly dangerous weapons, drugs, poisons, etc. can also be info hazards.
It's honestly not that hard to imagine real-life examples, without having the dangerous info in question or descending into government assassin-type scenarios.
It's just any data that, if you make it generally available, will predictably result in bad things happening.
In some cases -- like revealing important security flaws -- a public disclosure may be a last resort if less harmful methods have failed, but it's likely not an ethical first resort in those cases.
"Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt.
Despite my mistake, I am happy to say that my readership has so far been amazingly good about not using my rhetoric as an excuse to bully or belittle others. (I want to single out Scott Alexander in particular here, who is a nicer person than I am and an increasingly amazing writer on these topics, and may deserve a part of the credit for making the culture of Less Wrong a healthy one.)"
If you're not sure where to start, a number of his writings have been received well on HN: https://hn.algolia.com/?query=slatestarcodex&sort=byPopulari...
http://lesswrong.com/r/discussion/lw/lx0/rationality_from_ai...