I have vague memories of a scene from the UK comedy show "The Thin Blue Line" where a bunch of bad -isms were followed by counter-examples of good -isms, but despite the episode itself being easy to find (the episode is called "Ism Ism Ism"), I can't easily find the scene within the episode.
I find myself in Cubism and realize I must have taken a wrong turn somewhere, but perspective has collapsed all around me and the way out is blocked by fragments of guitar strings and noses of long-dead Parisian mistresses.
My favourite part about the rationalists is that they are a completely normal community apart from the rationalism. You'd think a group of people devoted to being rational would be a group of monk-like beings, cerebral, disconnected from the concerns of the flesh.
Instead we get this weird hotbed of fanfiction, cults, wild sex crime allegations, financial schemes so brazen they almost wouldn't qualify as fraud and a vague sense of some sort of group that are capable of any evil. Plus I can't stop laughing at the idea of Aella as "that rationalist hooker" [0] - a combination of words which never fails to make me chuckle.
The whole scene really deserves some sort of film, book, video game or something. I can't get a vision out of my head where some straitlaced person realises rationalists are involved and their face twists in horror. Keeping an eye on them over the years the communities that grew up around LessWrong really are as good as any work of fiction. Plus they're doing their best to make the world better and maybe people will learn something about Bayesianism, who knows.
Have you ever seen “Jackie Brown”? There's a scene where a criminal shows his colleague the dead body he has in the trunk of his car and then proceeds to explain, in perfectly rational terms, why the guy in the trunk had to be killed.
I'll never forget that lesson. Rationality can be used to justify anything and everything. Invoking rationality as the basis for one's decisions or behavior doesn't mean anything by itself.
Rationality without morality is pretty dangerous, thats for sure, its trivial for a skilled smart manipulator to lead folks into pretty dark places while still feeling superior.
And then there's a non-trivial amount of properly crazy folks who think they are absolutely fine and behaving very rationally, their own version of reality and universe in their head is impenetrable. Psychiatrists have stories to tell, but either can't share or folks prefer not listening such things (same goes for most doctors, ambulance drivers etc.)
> completely normal community apart from the rationalism
I don't think widespread polyamory, group homes, radically different social norms around language use and conversation, or reorganizing your entire life around mitigating AI risk are "completely normal."
I spent years thinking its crazy to get your morals from a random book, but then I read this stuff and I understand the pitfalls of also having it all wide open. This idea we all define our morality works extremely poorly. I used to think it was a product of the old times of almost non existent education and honestly dumb "ancient world" people (lack of nutrition, lead poisoning...).
But I don't know, clearly smart and educated people can believe extremely dumb things. The smarter they are the weirder the setups they get themselves into. One almost comes to appreciate the existence of some religions with a history of many centuries to "chill out" and see what works, instead of any of the new age cults.
I’m not religious and open minded about new stuff but ultimately decided the ancient Stoics system of morals works perfectly for me- and is more time tested than even Christianity. I think the rationalists decision to use consequentialist morality in regular life is a huge mistake, and impossible to get right in practice. Humans cannot predict the future outcomes of our actions very well, but we can easily remember and follow a simple set of core values.
If you look at most great evil perpetrated by mankind, it's almost always someone trying to "do good". Sure there are serial killers or whatever, crimes of passion, but they just kill one or a few. The people who have killed millions, driven whole regions into famine and death, destroyed lives at an industrial scale.... they were 'doing good'.
Utilitarian consequentialism is the most evil of all philosophies. If someone is merely sadistic their sadism will be sated after some finite amount of abuse. If someone is greedy then their harm will at least be limited to what they can profit from and there is no profit in ruling over a cinder. But the harm possible by someone convinced that their actions are good in some abstract sense is without any bound.
The irony in the lesswrongers fixation on consequentialist morality is that their great fear of machine superintendence is derived from an expectation that its evil will arise from the same sorts of reasoning they engage in themselves. They simply fail to pause and ask "Are we the baddies?"
I usually leave it out of my complaints because they're so ineffectual that it's not a real threat. But the LW solution to "unaligned" machine intelligence is to create an AI god in their own image first, so that it can enslave humanity and all other lifeforms within its lightcone for their own best interest, and suppresses the creation of any competing "unaligned" God. So they're literally out to create the very thing they fear, but with the hubris to imagine that if it was theirs it would be "good". The ultimate fantasy of both the utilitarian consequentialist and most authoritarian mass murderers. If their doomsday fears come true my bet is that it will be at their own hand or that of their followers.
It's also the ultimate horseshoe for militant internet atheists-- "there is no God; and this is OK" becomes "there is no God; and we're gonna create one in our own image".
Interesting observation that the rationalists doomsday AI scenarios all are essentially just obvious horrific consequences of utilitarian consequentialism backed by unlimited power- yet they insist on trying to live by acting this way themselves.
> You'd think a group of people devoted to being rational would be a group of monk-like beings, cerebral, disconnected from the concerns of the flesh.
Human beings are fundamentally irrational, and it's a really great way to deceive oneself to believe otherwise.
If you can convince yourself that you're unbiased, objective and "rational", then whatever you happen to believe must be correct! You can rationalize practically anything, so it's no surprise rationalists start getting weird when left to their own devices.
“To do evil a human being must first of all believe that what he's doing is good... it is in the nature of the human being to seek a justification for his actions.” -Aleksandr Solzhenitsyn
“Man is not a rational animal; he is a rationalizing animal.” -Robert A. Heinlein
You know that what you are describing is the central observation of rationalism right ? "Humans rationalize, so you better be skeptical of your own beliefs".
You presenting it as a blind spot of rationalists, but that’s precisely the blind spot that rationalism warn about !
Philip K. Dick would eventually have written this as a darkly comedic novel. A rationalist death cult plans to unleash a mind-altering drug, meanwhile the protagonist discovers a possibly benevolent AI has just come online in a Berkeley campus basement, all set in an alternative California where Nixon is still president and we have mining colonies on Mars. (The drug disables the empathy center of your brain, but also reveals the true nature of Nixon, who turns out to be a simulacrum, as he was actually assassinated in 1963.)
Scandals with sex pests, minors kept in group houses, partnerships with ponzi scammers, orgies, grit sham charities that pocket the money, etc. isn't normal. It happens elsewhere, sure, but it's not normal.
It's gonzo.
But it's also not new, there are plenty of works of fiction with groups like this in them, perhaps inspired by actual experience with 1960s new wave and prior counterculture groups.
I love this, but I'm not sure it's right. As for cults, people want to join them. It's more like any sufficiently sophisticated ideology is hard to distinguish from reality.
Also both the cult founder and person who got shot in the beginning of the story are transwomen? Just started reading https://zizians.info ... Apparently this cult has a thing for transwomen, or influences them, or something. Lots of weirdness and I don't have time to read it all right now.
All very interesting and will probably be a movie someday.
It’s reasoning without reality, the same mocked with sayings like “arguing about how many angels can fit on the head of a pin.”
Reasoning must be checked against observation and experiment. It must be checked against reality.
It’s not just that humans are fallible and full of biases and failure modes. It’s that reason itself is only capable of crystalline perfection in domains where all priors can be enumerated and the system is closed, like pure math. Even there we know of mathematical constructs that exhibit phenomena like emergence and computational irreducibility where the future state can’t be guessed without fully evaluating the function; where “leaps” are provably impossible.
The craziest stuff is some of the longtermism stuff. They are literally writing sci fi (and cliche sci fi at that) and then reasoning from it as if it were real and using those conclusions to guide present day moral and political thinking. Absolute lunacy. We can’t predict the next 50 years let alone the next 5000.
Sometimes I think one of the evolutionary counter pressures that has likely prevented human IQ from being driven entirely up and to the right is that high IQ increases propensity for delusional thinking. I’ve met quite a few incredibly brilliant idiots. The smarter you are the more elaborate a prison you can construct for your own brain. The major innovation of the discipline of science (and it is a discipline) was to put forward a method to avoid this by taking a step, confirming, and only then taking the next step. It’s pretty simple but it requires restraint even when there’s a shiny thing that looks so “truthy” and cool.
> Sometimes I think one of the evolutionary counter pressures that has likely prevented human IQ from being driven entirely up and to the right is that high IQ increases propensity for delusional thinking.
I had often had this thought too. Imagine if humans were 100x more intelligent, would we have necessarily survived for 200k odd years? Society, technology, morality all evolve at different paces and if one “outcompetes” the others then an imbalance can happen and humanity could be wiped out.
Imagine if a caveman was super intelligent enough to build some type of weapon of mass destruction but humanity doesn’t yet have enough experience to police this person or have laws or power structures to keep themselves safe. If you consider language and “memetics” technology than you have to consider beings 100x smarter could also be master manipulators capable of leading the group off the ledge. so many ways civilization could be wiped out.
Really the only reason we are around today is that society as a whole’s ability to implement strategies at recognizing dangerous groups and individuals outpaces the individual or small group’s ability to wield technology for dangerous means (well at least up until now).
> The smarter you are the more elaborate a prison you can construct for your own brain.
Amazing one-sentence summary for so much of what is going wrong right now.
Brilliant, rich people trapped inside their own mind palaces, spending vast sums on remaking the world into something that matches their interior delusions.
Agreed, but a question occurred to me: what if high IQ people aren't more delusional but more capable of articulating or acting upon their delusions, thus making it more apparent?
The delusions of a normal intellect are probably less interesting, and they're less likely to be a person who garners attention generally. But they may be equally detached from reality.
I think it's mostly that they're more difficult to persuade that their delusions aren't simply them being smarter than the normies around them. That trait isn't unique to high IQ people or unique or people whose choice of reading material and social scene is based around the idea of accessing advanced modes of reasoning, but it likely is more common amongst them
High IQ in a modern test is about pattern recognition.
I haven't seen it pointed out yet, but oversensitive pattern recognition can lead to illusions.
I.e. if you didn't sleep for two days, you can start "seeing" weird stuff sometimes. It's cause you brain is hallucinating something that isn't there, but because you see something, brain recognizes pattern and alerts you before checking against other patterns and common sense.
So I'd imagine that if someone is very good at pattern recognition they could have very high IQ but also not have enough erudition to check if the patterns they see is actually there.
Which arguably what could have happened in Yudkowskis case, since he scored very high on IQ, but dropped out of middle school.
Logical conclusions are only as valid as your model. Whenever I see people praising their own logic, I hear "My axioms are so perfect that I'm not capable of questioning them, or even knowing what they are."
Absolutely. These folks always remind me of Robert Brandom's* idea of "formal logical inference" vs. "material logical inference." The former focuses only on the formal structure of an argument where the latter takes into account context and other variables.*
* I don't know if the idea originated with Brandom.
* Please, forgive the crude (partially wrong?) explanation.
Interesting to see CFAR involved. I'm a big fan of Julia Galef. She's the president and cofounder of CFAR.
One of the things that struck me most was an interview with her years ago talking about the people that CFAR let into their programs. She said something to the effect that they didn't let people in that were trying to change other people's minds, but that were out to clarify their own thinking.
I really liked that. I'm a big fan of intellectual honesty: pursuing truth (however loosey-goosey that is for humans) rather needing to be right or win the argument.
> they didn't let people in that were trying to change other people's minds, but that were out to clarify their own thinking.
I'm not sure how much we can take this at face value, given that the OP mentions a guy (Michael Vassar), who apparently associates himself with the rationalist space and has been accused of brainwashing people and driving them crazy with "mind tricks". There's even been allegations that the now infamous craziness of this Zizian group may effectively be downstream of that.
That could very well be true. However, it's still an important idea to me and, I think, a lot of others.
People and organizations change. It's possible the CFAR and its founders started out with one mission or goal and that changed internally for whatever reason.
I have heard a version saying that Vassar changed (after he started experimenting with drugs). And at some moment later, he also got banned from the community.
However, if you bracket out all the murders for a second (a big ask), I don't think there's anything particularly special about this cult.
This is just the modern incarnation of something that's been going on for almost a hundred years in California.
In California there are Bohemian social scenes with a mix of high-performing technical people, spiritual/New Age/woo people, and "human potential" secular spirituality and self-improvement.
Magicians and Satanists involved in the founding of JPL (with a cameo from L. Ron Hubbard.) in the 1940s and 1950s. EST seminars, Esalen, UFO movements, communal yoga groups. Lesser known is the "General Semantics" movement (really quite similar to the Rationalists) from the 1950s. All these movements mix and overlap.
Mostly these social scenes are weird but harmless. But genuine cults often form in them and then spin off.
The Rationalists themselves are just another incarnation of this. They may seem totally bizarre, but if you just dig back in the history of California, they look very familiar. They are part of this stew of self-improvement/spirituality/tech/meditation. In particular they are the secular self-improvement people -- one thing they did is run training seminars where they teach you new thought techniques that will help you be more productive and successful. Many of them also engage in Buddhist meditation practices or take psychedelic drugs for exploration. In that scene, cults form. There have been multiple other rationalist-adjacent cults, although those did not kill people thank God.
I think it's a mistake to start analyzing the rationalists' beliefs about Bayesianism and expected utility and all that, to try and wonder how a cult could have possibly formed in this social scene. Nothing should be less surprising.
Now that this cult went on to commit multiple murders -- this is surprising and frightening. Perhaps some of the rationalist beliefs have something to do with it, but since there are a small number of other cults that have committed murders, looking at shared factors across all of them, such as charismatic psychopaths as leaders, may be a better choice.
Edit: upon reading to the very very bottom of the article, the author makes the exact same point. I'll leave this up just in case others aren't good readers either.
I see a lot of parallels between the rationalists and the Beatniks- both seem to be a social experiment of radically throwing out the existing culture and lifestyle, and trying to think through something new from scratch.
In both cases there are a lot of failed experiments, and disastrous personal consequences for many of the individuals involved, but also a lot of discovery and good ideas.
It's odd. I really like the Moloch post. It's a super interesting story that clearly exemplifies a problem in the real world.
Well-intended people often end up acting against what's best because they only invest in the self, and don't act as a whole.
It's cool philosophy!
But um..
Taking it seriously as a sort of religious demon? You can characterize it for that purpose in the settings of stories, myths, theology etc, but in real-life? Nahh..
What the heck man
Go join a labor union or something instead, what the heck
Probably better to look and decide for yourself, but generally they teach specific techniques to reason more effectively in certain situations.
They introduced me to Bayesianism, which is something I now use extensively in my science career, and I appreciate specific strategies to avoid falling into cognitive biases, and to argue more effectively and honestly, such as steelmanning.
None of their lifestyle ideas really jive with me- I have no interest in things like raising children in a polyamorous group home, or in using consequentialist ethics combined with hand wavy mathematical models to make decisions in regular life.
Yea, was reading a biography of (JPL founder) Jack Parsons a few weeks ago, and was struck how similar the Zizian patter sounded to a lot of the Golden Dawn/proto-Scientology stuff.
I think it may have been inspired by this Norm joke.
Basically pointing out that the criminal act is what's actually the worst part and not just whatever is most annoying
Norm MacDonald : Now do you think Cosby's legacy will be hurt?
Jerry Seinfeld - Host : Yeah.
Norm MacDonald : You do, huh? I mean, there's a comedian, Patton Oswalt, he told me, "I think the worst part of the Cosby thing was the hypocrisy." And I disagreed.
Jerry Seinfeld - Host : You disagreed with that?
Norm MacDonald : Yeah, I thought it was the raping.
I've been following the "rationalist movement" for a long time, and there has always been so many early cult warning signs among the various sub groups. I'm sad that it's gone so far as to result in deaths and murder attempts with some of them. I expect it to get worse as some of the "rationalists" continue to embrace bizarre groupthink irrationality.
Yeah, Yudkowsky saying many years ago that it would be better for one person to be tortured for 50 years than for 3^^^3 people to get a dust speck in their eye was pretty much everything I needed to know to realize this group was going to be a problem. And things just get worse from there.
> saying many years ago that it would be better for one person to be tortured for 50 years than 3^^^3 people getting a dust speck in their eye
I just looked this up on lw and it is pretty slimy text.
These weasels always leave it ambiguous to leave room to deny if their perfect rational view somehow goes wrong. Notice instead of actually having a backbone or taking the stance he just says
> I think the answer is obvious. How about you?
I’m not a christian but this sharply contrast the story of christ in which he sacrificed himself to in his mind spare billions from torture and the worst anguish in the universe.
Here we see Yudkowsky casually implying that it is better for someone to get tortured to save the inconvenience of a speck of dust, but notice he doesn’t volunteer himself.
Its fundamental to these rationalist and effective altruists that they are the right ones and others must sacrifice to fulfill the rationalist elitist intelligent world view. Look at FTX and this group. Their point of view it is OK for others to suffer financially and it is morally justified for the “intelligent” to steal and kill because they are the right ones and the ends justify the means.
Yeah, he leaves the original post ambiguous, but if you dig into the comments, he clarifies that his stance is that torture is the right option. I'm going to try to find the link.
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
When you find yourself writing "torture is the obvious option", you should realize that something has gone deeply wrong with the way you view the world. It's no wonder that death cults have sprung out of this philosophy.
I feel like this is missing the point. Trolley problems are absurd hypotheticals with no direct bearing on reality. Their usefulness is as thought experiments.
If you don't pull the lever the trolley kills one person. If you pull it N people lose a single limb. At what value of N does your ethical framework place the crossover point and why?
I don't agree that they're absurd. They have widespread popularity because they capture the kind of cost dilemmas that crop up all the time, eg 'if we build this refinery we can predict that an additional 20 people will get cancer over the next decade, but otoh we might achieve a 2% reduction in the cost of gasoline for everyone in our state over the same period.'
The main intellectual error of utilitarianism is the assumption of perfect foreknowledge, and the justification of conclusions as if those predictions were facts. But just because trolley problems are stark oversimplifications does not make them useless as ethical reasoning tools; studying them helps you recognize messy ethical dilemmas as such rather than being seduced by the alleged upsides of a policy proposal.
Just to be a little pedantic: Your refinery example isn't a trolley problem, because there are more than two choices, and one of the possible choices is "don't build it in the first place". The refinery example allows foresight, which the trolley problem doesn't. "I would build the tracks differently" isn't a possible answer to the trolley problem. The whole point is that it's too late to make any other decision than "kill few" or "kill many".
It's way worse than that. It can be 3^^^3 people getting a dust speck in their eye that only exist in some counterfactual world that's a mere figment of your imagination. Nevertheless this can have real-world consequences because "counterfactual mugging" is a valid concern given Rationalist logic. Very confusing.
The Zizians have only a very tangential relationship with the rationalist community. Once the leaders of the rationalist community understood what the Zizians were about, they banned them from their gatherings and published warnings about them.
According to one source (which I can dig up on request) all 4 of the Zizian engaged in murder are trans. Do you also expect crimes from trans young people to get worse?
The Zizians are mainly vegan, and the intellectual leader of the group (Ziz) advocated the murder of meat eaters. Do you consequently expect crimes done by vegans to get worse?
The Zizians are anarchist leftists, ...
In sum, you don't give much of an justification ("embrace bizarre groupthink irrationality" is not much) for pinning this on the rationalists and not on these other groups.
It is quite humorous to me in a dark way how the preeminent ethics movement of our times seems to spawn the most detestable behavior. How many Bentham essays and LessWrong posts do you need to read to conclude that psychological manipulation and wonton murder do not in fact contribute to the wellbeing of the world? There is a certain personality for whom rationality takes over their whole being, and they lose all their ability to feel connected to others in a profound way, and at this point their behavior is simply derived from whatever is left after they do the math.
I don't have a way to prove this and it's based purely on intuition/experience, but I disagree. I think ideology doesn't meaningfully influence such people. There is a deeper psychological drive and they only create a philosophical justification later, which is of little importance. This is perhaps true of humans in general.
you're both correct. All humans have a hardware component, an ideology acts as an attractor. Humans finding connection with other humans like them is one of the strongest pulls for a human mind. It has great power to amplify & grow. This can go in a direction of thriving and growth, and "win win" with their environment, or it can go in a direction of hate and aggression
I see two outcomes usually in the rationalist community:
- those who can manipulate language enough to justify their behavior and can then manipulate others
- those who modify their behavior and usually have some kind of mental breakdown because of internal conflicts
To me it is a predator-prey abusive LARP that got out of scope and feeds on fragile people looking for sense in their lives. Just look at how new comers are hazed and tested when they come on their forums and discords and what not.
> To me it is a predator-prey abusive LARP that got out of scope and feeds on fragile people looking for sense in their lives. Just look at how new comers are hazed and tested when they come on their forums and discords and what not
Yeah I think this is exactly what it is. The most eloquent charlatans end up roping in people who are lost or looking for meaning.
It’s always interesting to think how much self awareness the leaders have. Do they drink their own kool-aid or they laughing to themselves while they pouring the next dose for the others?
Every single ethics movement in history ended up spawning utter and complete nope (as well as a lot of useful concepts). See christianity, the enlightment liberalism, marxism and so on. It is almost as if the idea of universal, objective and cognizable good is inherently evil.
- humans find useful concept X
- they describe it with label Y
- it's genuinely useful, it spreads
- It gets too big, Y is misunderstood and corrupted
- New group of humans rediscovers concept X, gives it label Z
This is the story of humanity. The good news is we're kind of (mostly) stumbling through a upwards spiral. Current religion would be unrecognizable to the people in ancient times. It was never meant to be something frozen in stone. Folklore and things changing as they're retold was a feature, not a bug.
There's a great write up on this [1], but TL;DR, religion is cultural technology. It succeeded in doing exactly what it tried to do at the time (get people to stop killing each other in tiny tribes and allow mass decentralized human coordination to build civilization & empires where humans could be safe from the elements of nature)
I think it's an apples and oranges comparison to lump in Christianity (a religion that outright predicts that people will abuse it and protects itself against that) with liberalism/marxism (a philosophy that has no such protection and can be mangled into whatever you want). If anything, liberalism/marxism are more like secularized offspring of Christianity, given that they would probably never developed if it wasn't for their founding figures living in a western moral context completely drenched in Christian ideas.
I think I would’ve used Objectivism as a contrasting example. It’s designed around the idea that whatever a “strong” person does to fulfill their goals is inherently good. Objectivists wouldn’t phrase it that way, surely, but that seems the inevitable end result.
I'm not comparing them. What I'm saying is despite all the differences, the end result (dogmatic and self-righteous ideology with zero ability to align beliefs with reality) is eerily similar.
Curious how Christianity protects itself from abuse? From my perspective, Christianity is used to justify incredibly un-Christian activities pretty much constantly.
1. Christianity (like other religions) has built-in protections against false teaching within their own theology, which isn't really the same for secular philosophical frameworks.
2. There's a level of outlier visibility going on with a lot of people who abuse Christianity. The Christian who sincerely follows Jesus and walks in obedience don't seek out visibility or to exalt themselves. Even ones that evangelize do it on a local scale most often. Meanwhile people who abuse Christianity (Prosperity gospel, Christian nationalists, etc.) try to seek out large followings to bolster their power or wealth, making them seem like the "face" of Christianity whe
To elaborate, the idea is that a Christian (someone who has accepted Jesus as Lord and Savior) will show an outward transformation into someone who is Christ-like and obedient to God. When this doesn't happen at all, and that they remain completely un-Christian, you know it isn't genuine (See Matthew 7:15-20). The idea of how a Christian is shown by their outward renewal also touched on in Romans 12:9-21, Galatians 5:16-24, etc. It's not a perfect process, and it's the renewal is not a pre-requisite to salvation but rather an end result of salvation (Ephesians 2:8-12).
Therefore, Christians have a framework that can be used to identify and rebuke people who distort the teachings of Christianity into something that is in rebellion to God. This doesn't really exist with secular philosophy, which lays out the "ideal" but has no way to prevent itself from being warped.
Rationalism doesn't involve doing math, but _role-playing_ like you're doing math. There's lots of talk about updating priors and bayes; but in practice a lot of stuff isn't that quantifiable without running scientific studies, so this comes down to #yolo-ing it.
Of course thinking about stuff without always using formal statistics is fine and how people work, but if you trick yourself into thinking that you do use statistics for everything it may become harder to evaluate or second guess your own thinking.
Math LARPing is very accurate. I'm so sick of them talking about Bayes and "adjusting priors", because the way they use it is so loose that it can be used to justify literally anything.
I think they're pretty much ideologically opposed to serious research. They believe in pre-scientific "thinking from first principles", like Aristotle or something, using whatever data they can gather in a few minutes while avoiding any serious scholarship that would put the data in context, because then you're just an inside-the-box-thinking expert, rather than the freethinker with novel ideas that can only come when you don't know what you're talking about.
In large enough groups of people there'll always be some crazies. Are there really more of them to come out of LW circles than a comparably large other group?
I think describing them as “the preeminent ethics movement of our times” here is begging the question. Are people outside the group beyond we observers who periodically talk about them in places like this even aware they exist? Are there many philosophy grad students studying rationalism for their PhDs?
> how the preeminent ethics movement of our times seems to spawn the most detestable behavior
Excuse me if my sarcasm detector is faulty, but I wouldn't describe the rationality sphere like that. It's a niche group with a fetishism for 'intelligence', with a profound distaste for the liberal arts, which translates into a closed ecosystem of blog posts, themes and jargon, and a lack of reading actual books where they would see that they're retreading old stuff, but worse. It's a culture of people believing themselves immune to bias, where calling out obviously malicious behavior is 'not charitable' and all thought outside the group is suspect. Is it then strange that it produces people who just rationalize (forgive me) their bad impulses?
Most techies stopped learning about stuff not related to computers in high school, so yeah no wonder LessWrong, being more accessible, seems like a better option than reading actual philosophy.
Oh! Of course, but the I think the word that OP used then should've been influential. Preeminent is more an adjective of quality I believe, but it might be my ESL showing.
To be honest these seem like crazy people drawn to a movement that would have them and allow them to rise to prominence, rather than a movement creating crazy people.
I think many psychiatrists view split personality disorder as a largely nonexistent condition arising artificially from charlatan psychotherapists. It seems quite possible to me that these people drove themselves mad and each other to suicide with these experiments (where they expected to evoke a split personality they had already begun to define).
Sure this "Unihemispheric sleep" stuff cannot have helped any pre-existing issues, but I think we shouldn't discount the existence of those issues which predate what amounts to cult indoctrination.
The human mind is quite malleable. You can make yourself mentally ill, or be made mentally ill by the people around you. This fact seems to be under-acknowledged because there is a slippery slope to blaming the victims.
But just because you might have gotten yourself into something that doesn't mean you can get yourself out of it.
I'm skeptical of people who think they wouldn't be vulnerable. But even if some are not, -- some are, and you can start with the seeds of a little crazy and heal it through grounded thinking and healthy practices, or you can blow it wide open with crazed thinking, casual psychedelic use, and abusive cult practices.
I have had zero real-life interaction with the rationalist space beyond exchanging some emails with people and reading numerous blog posts, reading lesswrong, and stuff like that, but this post seems to lean too far into drawing a conclusion about a very non-homogeneous group of people based on a few individuals that seem to be mentally unwell.
While I have become less enthusiastic about rationalism (in the sense of this post, not in the sense of the philosophical tradition) over time, the space seems to consist mostly of nerdy people who try really hard to reason properly. Sure, they sometimes come to absurd conclusions, but in general they have interesting perspectives and analyses. Without data on the percentage of people who reach the cult-like status referenced in the post, it seems premature to deride the entire enterprise; in any sufficiently large population group you will have that type of behavior.
On the other hand the point about them being related in spirit to the various 'self-improvement', 'New Age', or whatever movements in California surely isn't WRONG, but it feels a bit... underdeveloped? I'm sure there are loads of connections, simply because the movement seems to be centered (in meatspace, anyway) in California. A more scholarly analysis explicitly drawing the connections would be more useful (and interesting!) than an ad hominem comparison to Scientology and the Manson family(!).
There are different subgroups within the rationalist movement, just as there are different subgroups within many organized religions. It’s totally normal to have a movement with plenty of generally normal people, and also to have a committed and very opinionated subgroup within that movement that has access to resources and power. You might think of Scientology or the Catholic Church as examples of organizations where this is made very explicit, but it can happen organically as well. The main thing that makes the rationalist/EA groups worrying, in my opinion, is the massive access to money and resources that also tends to be concentrated in the hands of the people with the strongest beliefs (see TFA or, more importantly, the AI “doomer” folks.) We’re talking about groups of smart people who genuinely fear the world is going to end, have convinced themselves that stopping it is critical to saving future humanity, in some cases have access to capable scientists, and most critically: have resources ranging into the 10s and 100s of million dollars. This is potentially a very dangerous situation if the folks in charge make bad decisions.
Zizians are a subgroup of Rationalists in the same sense as Satanists are a subgroup of Christians, or Scientology a subgroup of Psychiatry.
Even the article says that Ziz has attended a Rationalist workshop... and was told that she was a "net negative". Then she (and her followers) spent the following years fighting against the Rationalist community in various ways: anonymously accusing its members of rape, organizing protests, calling them evil on her blog, etc.
And yet some people keep writing as if both are more or less the same thing, which is something that both groups would strongly disagree with.
Who are really big into talking about reasoning with formal tools which few of them make any effort to understand or use.
They might as well "try really hard to reason properly" with crystals and burning sage. I think actually I'd trust the results of a crystal-reasoner more... at least they're more likely to have a bit of self-doubt that the crystals are all nonsense.
Go ask one of them -- one without immediate access to an LLM-- what the conjugiate prior is for observations of a binomial trial (or the same question in lay language). You'll get blank stares, yet this should be a bread and butter question for exactly the kind of reasoning they (claim to) advance.
There is a level of mathematical reasoning that basically anyone can learn, such as multiplying payoff matrices with probabilities and getting expected values. But going beyond that (e.g. adding uncertainties and correlations, or reasoning backwards from observations to model parameters) being extremely mathematically complex very fast, and rapidly unreasonable to apply to most real situations. Even if you're comfortable with the required graduate level mathematics it just takes too much time. Maybe someone could make a handbook of shortcuts to do the math for many common cases, it's the sort of thing you might expect a "rationalist group" to do, but instead they spend their time making excuses for fringe edgelord politics disguised with a bunch of inapplicable formal language. Gotta eugenics the plebs for their own good and if you don't agree it must be because you reject MATH.
They use bayesian this and that as magic words and to put on a show of intellectual superiority. But for the most part they can't and don't use these techniques, and for perfectly good reasons-- too hard, pointless without better data, etc. But they endlessly talk about them. The whole community is a mix of larpers and people who feel inferior because they don't know the other people are larping.
In their original form, largely harmless larpers... other than the rejection of "cognitive biases" resulting in occasionally promoting gonzo policies and grifting/embezzling nonprofit monies (E.g. SIAI). But as of recent years they've metastasized into an anti-AI apocalypse cult. Their founder has been out in the media advocating the use of nuclear weapons against civilian populations to suppress AI development, so we should just be entirely unsurprised that there is a radical offshoot group out there murdering people.
Hm. So I think if you define reasoning broadly -- as in collecting relevant data, enumerating and raking priorities, thinking through expected consequences... people do engage in this quite often in all areas of life, far more often than lesswrongers would credit.
If you define it more narrowly, as in using particular techniques that are popular to talk about in that community like bayes theorem, updating priors and so on-- then indeed it's not common but it's also not practiced by practically any of the community members.
I would say that there is a mix of both definitions in use in those communities, but for the former people already do it often and for the latter not even the lesswrongers do (with a few exceptions).
The fundamental problem with rationalism and just using pure reason and logic in general is that logic has no (necessary) connection to reality. It's a game played with symbols, and how you choose to map concepts and things in reality to those symbols determines your conclusion. There's a lot of room to setup your predicates in a way that justifies any conclusion you care to reach, no matter how monstrous, and there are lots of ways to turn remote probabilities into near-certainties. Frequently where they go wrong is injecting infinities or near-infinities into their calculations and once you put the lives of billions of people up against one person almost anything is justified.
The risk of giving yourself "split personalities" should not be underestimated.
I recently had a mild panic attack in which I became convinced that my subconscious was secretly working against me. It was terrifying, and I couldn't see a way to think myself out of it. In fact I thought the very fact that I was worrying about it proved that my subconscious had planted the idea in my conscious experience specifically to hurt me.
What worked was going for a short walk and physically touching the ground with my hands. "Touch grass" actually works sometimes, I think because if the stimuli of the mind are coming from within, then you have no way to override bad ones and you just get echoes of the same negative thing over and over again. Whereas if you can get stimulus of any kind from the external world, then you have something else to pay attention to, you can turn the focus outwards and the feedback loop subsides.
I no longer believe my subconscious is working against me. But I agreed to try to pay attention to its concerns and take them more seriously in the future, so that it has no reason to work against me.
This is no diss or slam, but if you are concerned about the inner workings of your mind working against you, I highly recommend some additional professional help to peel apart the onion a little bit more.
Concur. Also concur on external stimulus, and I'm pretty sure it works for exactly the reason you identified. I find the intrusive thoughts show up in the evening precisely because there's less stimulus (inside and out; my brain is quieting itself for sleep, giving the nastier, weirder parts of it their chance to say just the stupidest shit "out loud").
A major eye-opener for me was learning, awhile ago, that we differ from some other animals in that our limbic system is completely "wrapped" in malleable neurons. Some animals have (as far as we can tell) implastic wiring leading from some stimulus constructs straight into that system: they receive a stimulus, they respond immediately with physiological, emotional change. Ours is far more deeply tied to the plastic layers of the brain; we have a huge capacity to learn very complex stimulus-response ties between the world around us and how we should set our emotional state.
On the one hand, that's great! It's good to think that we can change our responses to stimuli in a way that, say, penguins can't.
On the other hand... "plastic" doesn't mean "you can will yourself to change them willy-nilly." It's more that we're capable of developing "mental allergies." We are unusual in that we can develop things like PTSD: you were shot at the same time you saw a Volkswagen Beetle drive by? Congratulations, you now experience a visceral survival-focused reaction every time you see one of the most classic examples of German engineering, and you can't will that feeling to stop because your brain's physiology changed under the dim hope that somehow avoiding vaguely-round cars avoids bullets, too.
Our brains are wonderful, transcendental things, but they're also machines and they're machines that can malfunction. It's good we have professionals who study this.
If you regularly struggle with your subconscious working against you, I recommend combat sports or rock climbing.
When there’s a punch coming straight for your face or you look down to see a long fall below you your mind has little choice but to pull itself together.
They tried the “hack” on one person and they apparently had a breakdown and killed themselves. Just a reminder not to take tips from “vegan Sith” who ended up rationalizing stabbing people as “double good” or some whacky shit like that.
Lots of animals are known to do this. Animals researchers wondered how marine mammals were able to sleep. Wouldn't you drown? Turns out dolphins sleep with half their brain awake, so they can surface and breathe. The brain hemispheres then take turns.
Certain species of ducks also sleep with half the brain. They get in a circle, with the awake eye facing out, and the sleeping eye in. (I guess if they have cross visual cortex nerves like we do, it means the brain hemisphere facing out is sleeping).
The murders are very sad but the pseudo science here has some humor to it.
Like they take the fact that the brain really has two hemispheres, and make some wildly unscientific claim that this means each person has two beings within it that can be shut off at different times (one side may be good and the other bad, one side may be female and the other male).
I honestly don't see how anyone can into this stuff unless they are on drugs or malnourished.
This idea of "Rationalism" does bear a remarkable correspondence to its 17th century cousin, in that it allows "Reason" to run wild, to go to the end of all its conclusions. That is the importance of Kant's Critique of Pure Reason, as Kant introduces finitude into the discourse of philosophy; since, Reason, left to run wild, will always miss the empirical world. Thus, there can be stupid questions, there can be things that are just "nuts," not because they can be proven right or wrong, but because the very basis of the question is already outside of empirical possibility and practical use.
What's particularly scary about this cult, when compared to others, is how some of their weird ideas permeate to those in powers. Tech oligarchs speak of "long-termism", "accelerationism", etc. They use these concepts to undermine worker rights and justify increasing inequalities, all in a vague utilitarian perspective of future utopia.
The funniest thing to me about “rationalists” and “less wrong” is the implication that other schools of thought are “irrationalist” or “more wrong”. The smugness comes before anything else.
They’re really invested in the study of philosophy, epistemology, and ethics and how it applies to their lives, and frequently reference philosophers and ethicists.
Half the time I come across that community online, it’s just love triangles between people with anime profile pictures, “rationalizing” atrocities, and comparing IQ test scores.
You average grad student in physics is less wrong than Newton in physics (he knows about special relativity for example). It’s not "smugness", of course building on the shoulders of those who comes before means building better than them.
But that’s the thing— they are quick to discard those ideas when confronted with the promises of technology and AI. As if glorified Markov chains are somehow going to alter our essential humanity. See the trans-humanists, or that guy who injects his son’s blood.
Ironically, you exemplify the standard of discourse that the rationalist spaces of discussion like lesswrong strives to go above, which is what makes them so valuable.
Loaded question (well, affirmation), on a confident sneery tone while using poorly defined terms ("essential humanity") and very poor comprehension of object-level facts of the ground ("Markov chains"). And no sign of even trying to understand the other side, just trying to score rhetoric points on a internet board.
> Ironically, you exemplify the standard of discourse that the rationalist spaces of discussion like lesswrong strives to go above, which is what makes them so valuable.
I guess I misunderstood then. It’s not just an Internet forum. It’s an internet forum with strict moderators. And it’s _very_ valuable.
With that level of, uh, following implications, "Democrats" are smug because they believe other schools of thoughts are pro-dictatorship, "feminists" are hateful because they believe anyone else is anti-women, "romanticists" are mad because they think everybody else hates love, etc.
It's possible to embrace a label without saying "everybody else is the literal opposite of that label".
> By the same token, the ability to dismiss an argument with a “that sounds nuts,” without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.
Aka as "common sense". It seems like these people make small errors in their base assumptions or reasoning. Over time, as these mistakes compound, they can lead to increasingly bizarre philosophies. Common sense might serve as a safeguard against such pitfalls, but it is frowned upon in those circles.
There are reasons, and the proponents aren't shy about stating them.
> Technological progress follows a sigmoid pattern not an exponential pattern.
That's true, but you don't know in advance where it will start to flatten.
> No significant new physics applicable to humans has been discovered since maybe the 1970s or 80s
It really depends on what you call "new physics". There are lots of discoveries related to quantum dots, graphene, high-temperature superconductivity, meta materials etc. Those aren't really "new" in the sense that they contradict the Schrödinger equation, but that was also true for Giant magnetoresistance, which was crucial for magnetic hard drives. Heck, a new form of magnetism was discovered just last year, first published in December 2024 (see https://www.nature.com/articles/s41586-024-08234-x for example).
Where in nature, outside supposedly, black holes do we see singularities? Humans and all human technology are a product of nature, why should we expect abstract "intelligence" resulting from human technology to accumulate into a singularity (unregulated positive feedback) instead of being subject to limiting constraints like every other natural phenomenon?
Belief in a singularity is the most irrational thing I have heard of. Supernatural things like the resurrection of Christ seem more likely than the Kurzweil/Yudkowski singularity.
The idea appeals to people who don't understand what intelligence is. I first heard of it and rejected it in high school more than 20 years ago.
Yudkowski didn't graduate highschool or go to college so he spent his time using his verbal skills to convince people he knew what he was talking about when in fact he is less of an authority in his field than even a moderately educated, intelligent person.
The fact this stuff continues to appeal to silicon valley types just goes to show they are not in fact much different from populist conspiracy believers in fly over country.
It is kind of funny actually to me, especially how these folks call themselves "rationalists..."
213 comments
[ 43.5 ms ] story [ 1722 ms ] threadIf it means you are unable to make any decisions or reach any conclusions which do not align with your -ism, then you're in a dead end.
Instead we get this weird hotbed of fanfiction, cults, wild sex crime allegations, financial schemes so brazen they almost wouldn't qualify as fraud and a vague sense of some sort of group that are capable of any evil. Plus I can't stop laughing at the idea of Aella as "that rationalist hooker" [0] - a combination of words which never fails to make me chuckle.
The whole scene really deserves some sort of film, book, video game or something. I can't get a vision out of my head where some straitlaced person realises rationalists are involved and their face twists in horror. Keeping an eye on them over the years the communities that grew up around LessWrong really are as good as any work of fiction. Plus they're doing their best to make the world better and maybe people will learn something about Bayesianism, who knows.
[0] https://knowingless.com/about/
I'll never forget that lesson. Rationality can be used to justify anything and everything. Invoking rationality as the basis for one's decisions or behavior doesn't mean anything by itself.
"That's Beaumont."
"Who's Beaumont?"
"An employee I had to let go."
The rest of the perfectly rational explanation will be left out, as it contains strong language not suitable for a website such as this.
And then there's a non-trivial amount of properly crazy folks who think they are absolutely fine and behaving very rationally, their own version of reality and universe in their head is impenetrable. Psychiatrists have stories to tell, but either can't share or folks prefer not listening such things (same goes for most doctors, ambulance drivers etc.)
I don't think widespread polyamory, group homes, radically different social norms around language use and conversation, or reorganizing your entire life around mitigating AI risk are "completely normal."
I'll give the rationalists that: they were out talking about AI ethics and risk way before AI risk was cool. Total hipsters.
But I don't know, clearly smart and educated people can believe extremely dumb things. The smarter they are the weirder the setups they get themselves into. One almost comes to appreciate the existence of some religions with a history of many centuries to "chill out" and see what works, instead of any of the new age cults.
Utilitarian consequentialism is the most evil of all philosophies. If someone is merely sadistic their sadism will be sated after some finite amount of abuse. If someone is greedy then their harm will at least be limited to what they can profit from and there is no profit in ruling over a cinder. But the harm possible by someone convinced that their actions are good in some abstract sense is without any bound.
The irony in the lesswrongers fixation on consequentialist morality is that their great fear of machine superintendence is derived from an expectation that its evil will arise from the same sorts of reasoning they engage in themselves. They simply fail to pause and ask "Are we the baddies?"
I usually leave it out of my complaints because they're so ineffectual that it's not a real threat. But the LW solution to "unaligned" machine intelligence is to create an AI god in their own image first, so that it can enslave humanity and all other lifeforms within its lightcone for their own best interest, and suppresses the creation of any competing "unaligned" God. So they're literally out to create the very thing they fear, but with the hubris to imagine that if it was theirs it would be "good". The ultimate fantasy of both the utilitarian consequentialist and most authoritarian mass murderers. If their doomsday fears come true my bet is that it will be at their own hand or that of their followers.
It's also the ultimate horseshoe for militant internet atheists-- "there is no God; and this is OK" becomes "there is no God; and we're gonna create one in our own image".
Human beings are fundamentally irrational, and it's a really great way to deceive oneself to believe otherwise.
If you can convince yourself that you're unbiased, objective and "rational", then whatever you happen to believe must be correct! You can rationalize practically anything, so it's no surprise rationalists start getting weird when left to their own devices.
“To do evil a human being must first of all believe that what he's doing is good... it is in the nature of the human being to seek a justification for his actions.” -Aleksandr Solzhenitsyn
“Man is not a rational animal; he is a rationalizing animal.” -Robert A. Heinlein
You presenting it as a blind spot of rationalists, but that’s precisely the blind spot that rationalism warn about !
It's gonzo.
But it's also not new, there are plenty of works of fiction with groups like this in them, perhaps inspired by actual experience with 1960s new wave and prior counterculture groups.
Also both the cult founder and person who got shot in the beginning of the story are transwomen? Just started reading https://zizians.info ... Apparently this cult has a thing for transwomen, or influences them, or something. Lots of weirdness and I don't have time to read it all right now.
All very interesting and will probably be a movie someday.
Reasoning must be checked against observation and experiment. It must be checked against reality.
It’s not just that humans are fallible and full of biases and failure modes. It’s that reason itself is only capable of crystalline perfection in domains where all priors can be enumerated and the system is closed, like pure math. Even there we know of mathematical constructs that exhibit phenomena like emergence and computational irreducibility where the future state can’t be guessed without fully evaluating the function; where “leaps” are provably impossible.
The craziest stuff is some of the longtermism stuff. They are literally writing sci fi (and cliche sci fi at that) and then reasoning from it as if it were real and using those conclusions to guide present day moral and political thinking. Absolute lunacy. We can’t predict the next 50 years let alone the next 5000.
Sometimes I think one of the evolutionary counter pressures that has likely prevented human IQ from being driven entirely up and to the right is that high IQ increases propensity for delusional thinking. I’ve met quite a few incredibly brilliant idiots. The smarter you are the more elaborate a prison you can construct for your own brain. The major innovation of the discipline of science (and it is a discipline) was to put forward a method to avoid this by taking a step, confirming, and only then taking the next step. It’s pretty simple but it requires restraint even when there’s a shiny thing that looks so “truthy” and cool.
I had often had this thought too. Imagine if humans were 100x more intelligent, would we have necessarily survived for 200k odd years? Society, technology, morality all evolve at different paces and if one “outcompetes” the others then an imbalance can happen and humanity could be wiped out.
Imagine if a caveman was super intelligent enough to build some type of weapon of mass destruction but humanity doesn’t yet have enough experience to police this person or have laws or power structures to keep themselves safe. If you consider language and “memetics” technology than you have to consider beings 100x smarter could also be master manipulators capable of leading the group off the ledge. so many ways civilization could be wiped out.
Really the only reason we are around today is that society as a whole’s ability to implement strategies at recognizing dangerous groups and individuals outpaces the individual or small group’s ability to wield technology for dangerous means (well at least up until now).
Amazing one-sentence summary for so much of what is going wrong right now.
Brilliant, rich people trapped inside their own mind palaces, spending vast sums on remaking the world into something that matches their interior delusions.
The delusions of a normal intellect are probably less interesting, and they're less likely to be a person who garners attention generally. But they may be equally detached from reality.
I haven't seen it pointed out yet, but oversensitive pattern recognition can lead to illusions. I.e. if you didn't sleep for two days, you can start "seeing" weird stuff sometimes. It's cause you brain is hallucinating something that isn't there, but because you see something, brain recognizes pattern and alerts you before checking against other patterns and common sense.
So I'd imagine that if someone is very good at pattern recognition they could have very high IQ but also not have enough erudition to check if the patterns they see is actually there.
Which arguably what could have happened in Yudkowskis case, since he scored very high on IQ, but dropped out of middle school.
* I don't know if the idea originated with Brandom. * Please, forgive the crude (partially wrong?) explanation.
I've been saying this for years in varrious forms. Spot on.
One of the things that struck me most was an interview with her years ago talking about the people that CFAR let into their programs. She said something to the effect that they didn't let people in that were trying to change other people's minds, but that were out to clarify their own thinking.
I really liked that. I'm a big fan of intellectual honesty: pursuing truth (however loosey-goosey that is for humans) rather needing to be right or win the argument.
I'm not sure how much we can take this at face value, given that the OP mentions a guy (Michael Vassar), who apparently associates himself with the rationalist space and has been accused of brainwashing people and driving them crazy with "mind tricks". There's even been allegations that the now infamous craziness of this Zizian group may effectively be downstream of that.
(An earlier HN discussion is at https://news.ycombinator.com/item?id=42849281 )
People and organizations change. It's possible the CFAR and its founders started out with one mission or goal and that changed internally for whatever reason.
However, if you bracket out all the murders for a second (a big ask), I don't think there's anything particularly special about this cult.
This is just the modern incarnation of something that's been going on for almost a hundred years in California.
In California there are Bohemian social scenes with a mix of high-performing technical people, spiritual/New Age/woo people, and "human potential" secular spirituality and self-improvement.
Magicians and Satanists involved in the founding of JPL (with a cameo from L. Ron Hubbard.) in the 1940s and 1950s. EST seminars, Esalen, UFO movements, communal yoga groups. Lesser known is the "General Semantics" movement (really quite similar to the Rationalists) from the 1950s. All these movements mix and overlap.
Mostly these social scenes are weird but harmless. But genuine cults often form in them and then spin off.
The Rationalists themselves are just another incarnation of this. They may seem totally bizarre, but if you just dig back in the history of California, they look very familiar. They are part of this stew of self-improvement/spirituality/tech/meditation. In particular they are the secular self-improvement people -- one thing they did is run training seminars where they teach you new thought techniques that will help you be more productive and successful. Many of them also engage in Buddhist meditation practices or take psychedelic drugs for exploration. In that scene, cults form. There have been multiple other rationalist-adjacent cults, although those did not kill people thank God.
I think it's a mistake to start analyzing the rationalists' beliefs about Bayesianism and expected utility and all that, to try and wonder how a cult could have possibly formed in this social scene. Nothing should be less surprising.
Now that this cult went on to commit multiple murders -- this is surprising and frightening. Perhaps some of the rationalist beliefs have something to do with it, but since there are a small number of other cults that have committed murders, looking at shared factors across all of them, such as charismatic psychopaths as leaders, may be a better choice.
Edit: upon reading to the very very bottom of the article, the author makes the exact same point. I'll leave this up just in case others aren't good readers either.
In both cases there are a lot of failed experiments, and disastrous personal consequences for many of the individuals involved, but also a lot of discovery and good ideas.
Both groups are even obsessed with Moloch (https://slatestarcodex.com/2014/07/30/meditations-on-moloch/).
Well-intended people often end up acting against what's best because they only invest in the self, and don't act as a whole.
It's cool philosophy!
But um..
Taking it seriously as a sort of religious demon? You can characterize it for that purpose in the settings of stories, myths, theology etc, but in real-life? Nahh..
What the heck man
Go join a labor union or something instead, what the heck
They introduced me to Bayesianism, which is something I now use extensively in my science career, and I appreciate specific strategies to avoid falling into cognitive biases, and to argue more effectively and honestly, such as steelmanning.
None of their lifestyle ideas really jive with me- I have no interest in things like raising children in a polyamorous group home, or in using consequentialist ethics combined with hand wavy mathematical models to make decisions in regular life.
Seriously, that's the best opening sentence I have read in a long while, kudos!
I just looked this up on lw and it is pretty slimy text.
These weasels always leave it ambiguous to leave room to deny if their perfect rational view somehow goes wrong. Notice instead of actually having a backbone or taking the stance he just says
> I think the answer is obvious. How about you?
I’m not a christian but this sharply contrast the story of christ in which he sacrificed himself to in his mind spare billions from torture and the worst anguish in the universe.
Here we see Yudkowsky casually implying that it is better for someone to get tortured to save the inconvenience of a speck of dust, but notice he doesn’t volunteer himself.
Its fundamental to these rationalist and effective altruists that they are the right ones and others must sacrifice to fulfill the rationalist elitist intelligent world view. Look at FTX and this group. Their point of view it is OK for others to suffer financially and it is morally justified for the “intelligent” to steal and kill because they are the right ones and the ends justify the means.
EDIT: https://www.lesswrong.com/posts/3wYTFWY3LKQCnAptN/torture-vs...
> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
When you find yourself writing "torture is the obvious option", you should realize that something has gone deeply wrong with the way you view the world. It's no wonder that death cults have sprung out of this philosophy.
If you don't pull the lever the trolley kills one person. If you pull it N people lose a single limb. At what value of N does your ethical framework place the crossover point and why?
The main intellectual error of utilitarianism is the assumption of perfect foreknowledge, and the justification of conclusions as if those predictions were facts. But just because trolley problems are stark oversimplifications does not make them useless as ethical reasoning tools; studying them helps you recognize messy ethical dilemmas as such rather than being seduced by the alleged upsides of a policy proposal.
2008-10-14 https://www.lesswrong.com/posts/K9ZaZXDnL3SEmYZqB/ends-don-t...
According to one source (which I can dig up on request) all 4 of the Zizian engaged in murder are trans. Do you also expect crimes from trans young people to get worse?
The Zizians are mainly vegan, and the intellectual leader of the group (Ziz) advocated the murder of meat eaters. Do you consequently expect crimes done by vegans to get worse?
The Zizians are anarchist leftists, ...
In sum, you don't give much of an justification ("embrace bizarre groupthink irrationality" is not much) for pinning this on the rationalists and not on these other groups.
To me it is a predator-prey abusive LARP that got out of scope and feeds on fragile people looking for sense in their lives. Just look at how new comers are hazed and tested when they come on their forums and discords and what not.
Yeah I think this is exactly what it is. The most eloquent charlatans end up roping in people who are lost or looking for meaning.
It’s always interesting to think how much self awareness the leaders have. Do they drink their own kool-aid or they laughing to themselves while they pouring the next dose for the others?
- humans find useful concept X - they describe it with label Y - it's genuinely useful, it spreads - It gets too big, Y is misunderstood and corrupted - New group of humans rediscovers concept X, gives it label Z
This is the story of humanity. The good news is we're kind of (mostly) stumbling through a upwards spiral. Current religion would be unrecognizable to the people in ancient times. It was never meant to be something frozen in stone. Folklore and things changing as they're retold was a feature, not a bug.
There's a great write up on this [1], but TL;DR, religion is cultural technology. It succeeded in doing exactly what it tried to do at the time (get people to stop killing each other in tiny tribes and allow mass decentralized human coordination to build civilization & empires where humans could be safe from the elements of nature)
[1] https://defenderofthebasic.substack.com/p/a-beginners-guide-...
2. There's a level of outlier visibility going on with a lot of people who abuse Christianity. The Christian who sincerely follows Jesus and walks in obedience don't seek out visibility or to exalt themselves. Even ones that evangelize do it on a local scale most often. Meanwhile people who abuse Christianity (Prosperity gospel, Christian nationalists, etc.) try to seek out large followings to bolster their power or wealth, making them seem like the "face" of Christianity whe
You are just saying the Bible has a bunch of text that lays out what being Christian is and isn't?
To elaborate, the idea is that a Christian (someone who has accepted Jesus as Lord and Savior) will show an outward transformation into someone who is Christ-like and obedient to God. When this doesn't happen at all, and that they remain completely un-Christian, you know it isn't genuine (See Matthew 7:15-20). The idea of how a Christian is shown by their outward renewal also touched on in Romans 12:9-21, Galatians 5:16-24, etc. It's not a perfect process, and it's the renewal is not a pre-requisite to salvation but rather an end result of salvation (Ephesians 2:8-12).
Therefore, Christians have a framework that can be used to identify and rebuke people who distort the teachings of Christianity into something that is in rebellion to God. This doesn't really exist with secular philosophy, which lays out the "ideal" but has no way to prevent itself from being warped.
Of course thinking about stuff without always using formal statistics is fine and how people work, but if you trick yourself into thinking that you do use statistics for everything it may become harder to evaluate or second guess your own thinking.
Excuse me if my sarcasm detector is faulty, but I wouldn't describe the rationality sphere like that. It's a niche group with a fetishism for 'intelligence', with a profound distaste for the liberal arts, which translates into a closed ecosystem of blog posts, themes and jargon, and a lack of reading actual books where they would see that they're retreading old stuff, but worse. It's a culture of people believing themselves immune to bias, where calling out obviously malicious behavior is 'not charitable' and all thought outside the group is suspect. Is it then strange that it produces people who just rationalize (forgive me) their bad impulses?
Most techies stopped learning about stuff not related to computers in high school, so yeah no wonder LessWrong, being more accessible, seems like a better option than reading actual philosophy.
https://en.wikipedia.org/wiki/Sybil_(Schreiber_book)
But just because you might have gotten yourself into something that doesn't mean you can get yourself out of it.
I'm skeptical of people who think they wouldn't be vulnerable. But even if some are not, -- some are, and you can start with the seeds of a little crazy and heal it through grounded thinking and healthy practices, or you can blow it wide open with crazed thinking, casual psychedelic use, and abusive cult practices.
While I have become less enthusiastic about rationalism (in the sense of this post, not in the sense of the philosophical tradition) over time, the space seems to consist mostly of nerdy people who try really hard to reason properly. Sure, they sometimes come to absurd conclusions, but in general they have interesting perspectives and analyses. Without data on the percentage of people who reach the cult-like status referenced in the post, it seems premature to deride the entire enterprise; in any sufficiently large population group you will have that type of behavior.
On the other hand the point about them being related in spirit to the various 'self-improvement', 'New Age', or whatever movements in California surely isn't WRONG, but it feels a bit... underdeveloped? I'm sure there are loads of connections, simply because the movement seems to be centered (in meatspace, anyway) in California. A more scholarly analysis explicitly drawing the connections would be more useful (and interesting!) than an ad hominem comparison to Scientology and the Manson family(!).
Even the article says that Ziz has attended a Rationalist workshop... and was told that she was a "net negative". Then she (and her followers) spent the following years fighting against the Rationalist community in various ways: anonymously accusing its members of rape, organizing protests, calling them evil on her blog, etc.
And yet some people keep writing as if both are more or less the same thing, which is something that both groups would strongly disagree with.
Who are really big into talking about reasoning with formal tools which few of them make any effort to understand or use.
They might as well "try really hard to reason properly" with crystals and burning sage. I think actually I'd trust the results of a crystal-reasoner more... at least they're more likely to have a bit of self-doubt that the crystals are all nonsense.
Go ask one of them -- one without immediate access to an LLM-- what the conjugiate prior is for observations of a binomial trial (or the same question in lay language). You'll get blank stares, yet this should be a bread and butter question for exactly the kind of reasoning they (claim to) advance.
There is a level of mathematical reasoning that basically anyone can learn, such as multiplying payoff matrices with probabilities and getting expected values. But going beyond that (e.g. adding uncertainties and correlations, or reasoning backwards from observations to model parameters) being extremely mathematically complex very fast, and rapidly unreasonable to apply to most real situations. Even if you're comfortable with the required graduate level mathematics it just takes too much time. Maybe someone could make a handbook of shortcuts to do the math for many common cases, it's the sort of thing you might expect a "rationalist group" to do, but instead they spend their time making excuses for fringe edgelord politics disguised with a bunch of inapplicable formal language. Gotta eugenics the plebs for their own good and if you don't agree it must be because you reject MATH.
They use bayesian this and that as magic words and to put on a show of intellectual superiority. But for the most part they can't and don't use these techniques, and for perfectly good reasons-- too hard, pointless without better data, etc. But they endlessly talk about them. The whole community is a mix of larpers and people who feel inferior because they don't know the other people are larping.
In their original form, largely harmless larpers... other than the rejection of "cognitive biases" resulting in occasionally promoting gonzo policies and grifting/embezzling nonprofit monies (E.g. SIAI). But as of recent years they've metastasized into an anti-AI apocalypse cult. Their founder has been out in the media advocating the use of nuclear weapons against civilian populations to suppress AI development, so we should just be entirely unsurprised that there is a radical offshoot group out there murdering people.
> > Who try really hard to reason properly
is not in my opinion the distinctive element of the group.
I think that the parent would have been more correct had he written
> who try really hard to apply reasoning to areas people often don't reason about.
As this implies an entirely different class of actions.
If you define it more narrowly, as in using particular techniques that are popular to talk about in that community like bayes theorem, updating priors and so on-- then indeed it's not common but it's also not practiced by practically any of the community members.
I would say that there is a mix of both definitions in use in those communities, but for the former people already do it often and for the latter not even the lesswrongers do (with a few exceptions).
I recently had a mild panic attack in which I became convinced that my subconscious was secretly working against me. It was terrifying, and I couldn't see a way to think myself out of it. In fact I thought the very fact that I was worrying about it proved that my subconscious had planted the idea in my conscious experience specifically to hurt me.
What worked was going for a short walk and physically touching the ground with my hands. "Touch grass" actually works sometimes, I think because if the stimuli of the mind are coming from within, then you have no way to override bad ones and you just get echoes of the same negative thing over and over again. Whereas if you can get stimulus of any kind from the external world, then you have something else to pay attention to, you can turn the focus outwards and the feedback loop subsides.
I no longer believe my subconscious is working against me. But I agreed to try to pay attention to its concerns and take them more seriously in the future, so that it has no reason to work against me.
A major eye-opener for me was learning, awhile ago, that we differ from some other animals in that our limbic system is completely "wrapped" in malleable neurons. Some animals have (as far as we can tell) implastic wiring leading from some stimulus constructs straight into that system: they receive a stimulus, they respond immediately with physiological, emotional change. Ours is far more deeply tied to the plastic layers of the brain; we have a huge capacity to learn very complex stimulus-response ties between the world around us and how we should set our emotional state.
On the one hand, that's great! It's good to think that we can change our responses to stimuli in a way that, say, penguins can't.
On the other hand... "plastic" doesn't mean "you can will yourself to change them willy-nilly." It's more that we're capable of developing "mental allergies." We are unusual in that we can develop things like PTSD: you were shot at the same time you saw a Volkswagen Beetle drive by? Congratulations, you now experience a visceral survival-focused reaction every time you see one of the most classic examples of German engineering, and you can't will that feeling to stop because your brain's physiology changed under the dim hope that somehow avoiding vaguely-round cars avoids bullets, too.
Our brains are wonderful, transcendental things, but they're also machines and they're machines that can malfunction. It's good we have professionals who study this.
When there’s a punch coming straight for your face or you look down to see a long fall below you your mind has little choice but to pull itself together.
Certain species of ducks also sleep with half the brain. They get in a circle, with the awake eye facing out, and the sleeping eye in. (I guess if they have cross visual cortex nerves like we do, it means the brain hemisphere facing out is sleeping).
https://en.wikipedia.org/wiki/Unihemispheric_slow-wave_sleep
So I guess if there's prior art, it's not a leap to think...maybe humans could try to do it too.
Like they take the fact that the brain really has two hemispheres, and make some wildly unscientific claim that this means each person has two beings within it that can be shut off at different times (one side may be good and the other bad, one side may be female and the other male).
I honestly don't see how anyone can into this stuff unless they are on drugs or malnourished.
Because make no mistake, these folks with the public benefit corporations doing doomer PR around AI are very, very much a branch of this family tree.
This idea of "Rationalism" does bear a remarkable correspondence to its 17th century cousin, in that it allows "Reason" to run wild, to go to the end of all its conclusions. That is the importance of Kant's Critique of Pure Reason, as Kant introduces finitude into the discourse of philosophy; since, Reason, left to run wild, will always miss the empirical world. Thus, there can be stupid questions, there can be things that are just "nuts," not because they can be proven right or wrong, but because the very basis of the question is already outside of empirical possibility and practical use.
Loaded question (well, affirmation), on a confident sneery tone while using poorly defined terms ("essential humanity") and very poor comprehension of object-level facts of the ground ("Markov chains"). And no sign of even trying to understand the other side, just trying to score rhetoric points on a internet board.
I guess I misunderstood then. It’s not just an Internet forum. It’s an internet forum with strict moderators. And it’s _very_ valuable.
It's possible to embrace a label without saying "everybody else is the literal opposite of that label".
https://youtu.be/h242eDB84zY?si=Q2Ef9Hi1kqMFdeNP
Aka as "common sense". It seems like these people make small errors in their base assumptions or reasoning. Over time, as these mistakes compound, they can lead to increasingly bizarre philosophies. Common sense might serve as a safeguard against such pitfalls, but it is frowned upon in those circles.
There are reasons, and the proponents aren't shy about stating them.
> Technological progress follows a sigmoid pattern not an exponential pattern.
That's true, but you don't know in advance where it will start to flatten.
> No significant new physics applicable to humans has been discovered since maybe the 1970s or 80s
It really depends on what you call "new physics". There are lots of discoveries related to quantum dots, graphene, high-temperature superconductivity, meta materials etc. Those aren't really "new" in the sense that they contradict the Schrödinger equation, but that was also true for Giant magnetoresistance, which was crucial for magnetic hard drives. Heck, a new form of magnetism was discovered just last year, first published in December 2024 (see https://www.nature.com/articles/s41586-024-08234-x for example).
Belief in a singularity is the most irrational thing I have heard of. Supernatural things like the resurrection of Christ seem more likely than the Kurzweil/Yudkowski singularity.
The idea appeals to people who don't understand what intelligence is. I first heard of it and rejected it in high school more than 20 years ago.
Yudkowski didn't graduate highschool or go to college so he spent his time using his verbal skills to convince people he knew what he was talking about when in fact he is less of an authority in his field than even a moderately educated, intelligent person.
The fact this stuff continues to appeal to silicon valley types just goes to show they are not in fact much different from populist conspiracy believers in fly over country.
It is kind of funny actually to me, especially how these folks call themselves "rationalists..."