I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.
"There are also some shared values between all these people" is guilt by association by definition.
The rest of your comment is even worse. Cults? Consider AGW cult where people reject any attempt of a discourse because "science is settled". Which is equivalent to "God told us this".
For those of you not familiar with the acronym, "AGW" stands for Anthropogenic Global Warming.
(Though, Istodd, please correct me if there is another meaning of which I am not aware. I would be embarrassed to have impugned your credibility in error.)
I mean they did claim that pointing to science to support your position in an argument is tantamount to dogmatic faith, so I don’t think you need to worry about impugning what isn’t there
This is the last reply you get because you’re already convinced you’re smarter than I am. Who knows, maybe you are, but disbelieving scientists sounds pretty stupid to me.
Science is peer reviewed by field experts who constantly test and develop said expertise in order to defeat their own biases, and their work backed by data. Scientists openly invite criticism and deconstruction of their work. That, nominally, is the whole point of getting your work published.
That’s a far cry from the dogmatic acceptance of faith that you’re pretending it is
It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.
The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
Didn’t the AI botnet (or whatever we should call them) from Anthropic try to get a PR with malware merged to an open source repo, and when the owner pushed back on the PR other agents created “fake humans” and commented on the PR that “No it’s completely fine, I’ve tested it”.
Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”.
Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive.
But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that.
What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
> What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI".
As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) that any folks in the groupthink bubble you are referring to.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations.
They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal
disagreements, and some systems are able to resolve them.
They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address.
What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
I'm talking specifically about discourse like Jacob Coxon's tweet thread and later communication. Those are the folks[1] you want to be listening to, not the nuts with the blogs, no matter how much a particular conspiracy confirms your priors.
Salvation will come in a whitepaper, not a screed.
So what you're saying is that we lack a reproduction study?
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons.
When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it.
The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural stories were always a bad hack.
I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
If you are serious about this preference, would you also be okay with a compromise solution where only you (and anyone else who feels the same) die, rather than taking everyone else with you?
If we all die because of AI, it will be because of the uses that people like Elon Musk and Peter Thiel and Palmer Luckey put it to, not because of a robot uprising.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
They're about as insular as the Unitarian church or Hacker News. It's basically an online community where anyone can check in, check out, and leave a comment or an essay. There's no formal structure or membership organization. The people who write or associate there could be edgy pre-teens or prominent math professors, it's really just a self-association thing. Some ideas have bubbled around more than others, like any memetic spread, but I don't think it's accurate to say that the people who associate there all agree on things, just like HN does not either. And nor is it any more of a clique than HN.
> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem with LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
I think the point is not so much that we can just ignore the possibility of AI doing harm, but that if AI really is the existential threat that these people claim, then they are the last people we should want as stewards of the technology. These people like to talk about AI ethics, but if the ethics of the AI they build is their ethics, as demonstrated by their actions, I want no part of it.
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
I really don’t know how we went from “there are people bullish about AI, and others that see it as an existential threat” to the complete reversal/psyop in the past few months where AI doomer has become a synonym of booster (?) and grouped with rationalist and other pro-AI weirdos.
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
I feel your frustration, but it’s just normal human behavior, no need for a psyop. A lot of people are emotionally invested in AI being a fad, useless, easily controlled, a capitalist boondoggle, a circular financing scam, a bubble about to explode. The alternative is that we live in an incredibly dangerous time, because obviously if we’re building something smarter than ourselves, we’re building the kitchen where we’ll be cooked.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
> when you can just start from zero, look at what's known to be happening right now
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
When the going gets weird, the weird turn pro. Few people have the ability/temperament to reason about things that have never happened before. Those that do have a weird psychological profile. Usually you can ignore them. Every once in a while, you can't.
I remember reading back in 2014 (IIRC) Yudkowsky's explanation of the Bayes theorem. It was genuinely good. Some sort of first principles introduction.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR was remarkable and I will always appreciate those works.
> Then the movement minted a fraud. Sam Bankman-Fried was not an accident of proximity; he was EA’s own product
TFA could have have mentioned that it's a fraud, MacAskill, who minted another fraud. MacAskill was SBF's guru and he was also involved in FTX and Alameda scams (but distanced himself before the scandal blew up) and MacAskill bought a 15 million GBP mansion in the UK for the Effective Altruism movement using funds stolen by SBF.
Last I checked the mansion hasn't been clawed back yet (but it could: funds from the Madoff ponzi have been clawed back more than 15 years after the fraud has been exposed).
Now of course people in charge of clawing back the stolen funds are the only hope of people who've been scammed for MacAskill didn't effectively altruistically sell the mansion to effectively altruistically hand the money to people who have been scammed.
That MacAskill and SBF maybe had groupsex with Caroline Ellison is one thing: they're proud to be polyamorous, whatever. We live in a free world.
But buying a $15 million mansion with stolen funds is thievery. Effective Scammers.
It seems that the author has very weird and strong obsession over Yudkowsky and his HP fanfic and took the public hatred toward AI companies as an opportunity to ridicule him.
> A sex worker from Idaho who became one of the subculture’s most-read writers and now runs an AI-doom propaganda residency with Grimes, Elon Musk’s ex-girlfriend.
I hate that I know who that is from this scant description.
> A splinter sect that fled to tugboats and whose members have been linked to six deaths. And a blogger who wanted to abolish democracy, found the ear of several billionaire patrons, and watched his ideas come to fruition in the vice president’s office.
I think I’m comfortable, even though the aspersion is on me, saying that anyone who can recognize Aella is too chronically online.
The zizians were a national headline and you only need to be plugged into US politics a little bit to know of Curtis yarvin given his influence on Peter thiel, musk, and Vance so I’d give knowledge of them a pass
Edit: I also know of the sanely diagram meme for her so that’s even more evidence. Maybe I should detox off the internet for a bit
Oh, man, not this again. I recommend listening to the "Behind the Bastards" podcast episodes on the Zizians if you want to really be confronted by the personal histories of some people who went really far down the rationalist rabbit hole. Yes, the harry potter fan fiction features prominently in it.
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
102 comments
[ 1.2 ms ] story [ 30.3 ms ] threadMore or less the same content but parent is much more palpable, IMHO.
[1] https://news.ycombinator.com/item?id=49737985
The rest of your comment is even worse. Cults? Consider AGW cult where people reject any attempt of a discourse because "science is settled". Which is equivalent to "God told us this".
(Though, Istodd, please correct me if there is another meaning of which I am not aware. I would be embarrassed to have impugned your credibility in error.)
but ye, who I am to persecute people based on their beliefs, this is for religious people to do.
Science is peer reviewed by field experts who constantly test and develop said expertise in order to defeat their own biases, and their work backed by data. Scientists openly invite criticism and deconstruction of their work. That, nominally, is the whole point of getting your work published.
That’s a far cry from the dogmatic acceptance of faith that you’re pretending it is
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”.
Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive.
But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that.
What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI".
As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."
This is the worst outcome.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
That’s like a running theme in these connected groups and even the article calls it out.
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
Solve one, and you solve the other.
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.
Salvation will come in a whitepaper, not a screed.
[1] And yes, that includes Amodei and Altman.
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
> AI will replace all of them soon.
What is that spaghetti of words you tied together to form a sentence? Were you trying to be comprehensible?
Their core belief system is morally abhorrent.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem with LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
Like, Yudkowski is a high school dropout who wrote a bunch of Harry Potter fanfic I cannot believe that people take the arguments he makes seriously.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR was remarkable and I will always appreciate those works.
https://www.yudkowsky.net/rational/virtues
> Then the movement minted a fraud. Sam Bankman-Fried was not an accident of proximity; he was EA’s own product
TFA could have have mentioned that it's a fraud, MacAskill, who minted another fraud. MacAskill was SBF's guru and he was also involved in FTX and Alameda scams (but distanced himself before the scandal blew up) and MacAskill bought a 15 million GBP mansion in the UK for the Effective Altruism movement using funds stolen by SBF.
Last I checked the mansion hasn't been clawed back yet (but it could: funds from the Madoff ponzi have been clawed back more than 15 years after the fraud has been exposed).
Now of course people in charge of clawing back the stolen funds are the only hope of people who've been scammed for MacAskill didn't effectively altruistically sell the mansion to effectively altruistically hand the money to people who have been scammed.
That MacAskill and SBF maybe had groupsex with Caroline Ellison is one thing: they're proud to be polyamorous, whatever. We live in a free world.
But buying a $15 million mansion with stolen funds is thievery. Effective Scammers.
No, it's because of an extensive and well coordinated FUD campaign. The resignation letter was just one salvo, and, like the rest, orchestrated.
I'm not a huge fan of calling LLMs "AI" and acting like they're our savior, but I'm even less of a fan of the brain-dead fearmongering.
I hate that I know who that is from this scant description.
> A splinter sect that fled to tugboats and whose members have been linked to six deaths. And a blogger who wanted to abolish democracy, found the ear of several billionaire patrons, and watched his ideas come to fruition in the vice president’s office.
Actually fuck I know who all of these people are.
Although perhaps we are excessively online
The zizians were a national headline and you only need to be plugged into US politics a little bit to know of Curtis yarvin given his influence on Peter thiel, musk, and Vance so I’d give knowledge of them a pass
Edit: I also know of the sanely diagram meme for her so that’s even more evidence. Maybe I should detox off the internet for a bit
https://www.google.com/search?client=firefox-b-d&q=behind+th...
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.