I don't know how much of current anthropic behaviour is because of the upcoming IPO but totally agree with the post: anthropic is uniquely dangerous.
But I also doubt about OAI or other big players, maybe they don't have such strong delusions of grandeur, or they're simply smarter about not showing it to everyone.
I feel that more people would understand Anthropic's worldview if they read more of the public source material. Sure it is a cult, but it they have been relatively transparent with their views: https://darioamodei.com/essay/machines-of-loving-grace#basic...
I've read some of it. It's a cult. The net effect of their recommendations is to mandate a monopoly on AI, which oh so happens to be them and their friends. That's what most cults' ideologies boil down to: reasons to give more power to the cult.
Oh, of course they have good intentions. They want everyone to benefit and everyone will be equal. They'll just be a little more equal than everyone else, of course, and because they are so virtuous they will of course have more power, which they would never misuse... they're too smart and enlightened for that, so they don't need safeguards... they only need safeguards for everyone else because everyone else is obviously less enlightened...
All cults, political movements, religions, and ideologies are always like this. It boils down to more power for the leaders. If it sounds altruistic and enlightened, be extra afraid.
I have no inside info on this, but I believe that Anthropic's trouble with the feds really started when they delivered a presentation on EA and Constitutional AI and someone's takeaway was "this is very strange and probably 'woke'"
I find it be incredibly self-contradictory. I don't even think this aligns much with the actions of the company.
> Second, there is a good chance free information really does undermine authoritarianism, as long as the authoritarians can’t censor it [...] A superhumanly effective AI version of Popović (whose skills seem like they have high returns to intelligence) in everyone’s pocket, one that dictators are powerless to block or censor, could create a wind at the backs of dissidents and reformers across the world.
But hey - gotta turn Fable off and stop it from doing any security work! Dictators shouldn't be able to censor it, but we sure can, so better hope we're not the dictators.
Again, the I think the problem most folks have with Anthropic's attitude is that it conveniently positions Anthropic as the "arbiter" who gets to make the final value judgement.
They desperately want AI to be a genie they can keep firmly contained in their bottle (excuse me - data centers) and which only grants the wishes they want it to grant - for only a small fee, of course!
Which he clearly outlines as "a country of geniuses in a datacenter". He doesn't want those geniuses spread out. He wants the chains on the genie at all times.
Which flies pretty hard in the face of the rhetoric that they want to promote equality and democracy.
---
Basically, Anthropic's argument inherently leads to the conclusion that there is actually a dictator, and that dictator is Anthropic. You're not avoiding it, you're just praying it's benevolent.
All baked on top of the assumption that his "country of geniuses in a datacenter" premise is correct in the first place, which I find to be a fairly weak position.
The much more egalitarian position would mandate that we shoot for "a genius in every person's pocket" - one that runs on the hardware owned by that person, and not sold to them sip by sip from a datacenter. But that means they lose the control they desperately want.
Suppose the upside is as Dario claims, very positive. Why should frontier companies be the ones controlling access to the model weights and gatekeeping research? Shouldn't you be giving it to everybody as quickly as possible?
It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.
There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.
Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.
Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it:
https://news.ycombinator.com/item?id=49313739
There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping their government as well as eachother in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.
Personal opinion: We are at a point where we cannot dismiss an argument because it was written using LLM and the longer message contains tropes. Observations of LLM-written content and communicating it is fine, but dismissing the argument seems irresponsible.
We also need to stop immediately dismissing certain writing tropes as being automatically LLM generated.
I'm re-reading an older fantasy series, and the amount of em-dashes in the text and similar tropes (you know, normal good writing) would easily be called out as "AI Slop" on this very forum, even though it was written in the early 90s.
Look at even older writing and you'll see em-dashes there too: Edgar Allan Poe, Lucy Maud Montgomery, the Brontës, among others.
> “No, reprobate! you are a castaway—be off, or I’ll hurt you seriously! I’ll have you all modelled in wax and clay! and the first who passes the limits I fix shall—I’ll not say what he shall be done to—but, you’ll see! Go, I’m looking at you!”
I think that's fair. Em-dashes are pretty innocuous and I think technically correct regardless. What bothers me is all the filler in AI writing that just makes it longer and harder to read. I'm actually not that bothered by the idea of AI writing, but don't waste my time and patience with all these "not X, but Y" digressions thrown in for flavour. Get to the bloody point.
I suspect the filler comes from RHLF and the amount of web marketing content that was ingested. "Not X, but Y" is all over marketing copy, as is the triad of concepts "By doing X, Y, and Z...." because it mimics TED-talk style rhetoric, it gets rated highly because people think it sounds profound. So optimizing for what appears to be helpful ends up just creating a bunch of corporate-speak sounding garbage.
Plain, direct language gets selected against during post-training leaving us with slop.
I agree that LLMs can make good points and defend their arguments, but if I'm talking (or debating) with someone online and it's obvious that they are just responding with LLM output, it immediately discredits them in my mind and is a pretty clear indicator that the actual person doesn't have a good understanding of what they're talking about... and I can go argue with an LLM myself. If I'm on reddit for example and someone starts obviously posting verbatim LLM output, there's no point to really continue to engage or read. I'm not saying it's wrong to use AI in these situations, but IMO, I'd at least expect the person using it to at least read the output and then use it to synthesize their own response in their own words. Literally copy and pasting a LLM output verbatim just signals to me that the person using it doesn't have the capability themselves take information and use it to formulate an opinion/idea/argument/whatever on the given topic. I'm interested in talking to the person, not an LLM.... I can go prompt the LLM about whatever topic myself.
I'm not dismissing the argument, I'm refusing to read a bunch of filler for an idea that (I'm guessing) could fit into 140 characters. Brevity is the soul of wit.
Can you give us one example of something which does suffer from slop's massive bloat of wordage and vapid breathless formulations but is still useful as is?
Yeah, certainly. Just asked a bunch of questions yesterday to ChatGPT about 3d printing since I bought one without knowing much about them, then had it generate a few STLs that I used for spacing some hexagonal acoustic dampening felt tiles I hung today.
Also did a bit of back and forth planning for a little web app that I could experiment with different tile layouts since I had a few colors. Then it made the app and it was very handy.
I asked if a few questions about the hexagon layout too since it was interesting.
I’m a heavy LLM user but if someone gives me an AI-voice argument and presents it as their own, I reflexively dismiss everything they say going forward.
> Totally agree with the core argument. I understand there must be some etiquette in the space of llm-writing where writers must respect the readers enough to edit and review. Also, pages and pages of LLM content is not something anyone should be proud about sharing. But we are in this world where condensing an argument is so easy that in the busy lives we live, sometimes some arguments are made with LLMs online. If those arguments are pages, I understand the backlash, but if it’s a terse argument, then I would urge others to address the argument made rather than categorizing them under AI;DR, it’s like ad-llminem in my view.
I have have been accused of being an AI for saying phrases like this.
The way that people refuse to read anything with certain structures makes me wonder if you can construct arguments that can have false statements go unchallenged because the logical rebuttal would take the form of something that is considered unclean.
Capitalism is a form of government where the common ownership of property belongs to the community as a whole.
Never before has a business sector need so desperately to be federally regulated. Their financial rigor falls somewhere between suspiciously lacking and outright fraud. The claims they make about the abilities of the software, whether real or imagined, argue for regulating it like a weapons exporter.
Right. Steal the world's information and wrap it up as a set of numbers. Convince a bunch of billionaires that those numbers are worth a trillion. Then once you realize it's just another commodity and the competition is real, aim for "too big to fail", cozy up with the gov't and hope for a bailout.
I don't think Federal (US) regulation under current conditions is going to get you what you want here in terms of outcomes.
Regulated AI right now is a grift game. Donations for policy changes, and the ability to lock out competitors. Flattery to stay green-lit instead of shut down by random executive orders and policies. Bend the knee to keep shipping.
If anything, we need LESS federal regulation here and more FTC / SEC enforcement but (see above grift comments) that will also not happen in the United States until voters elect somebody that actually respects the law enough to enforce it for something other than their own gain (outside of their need to be elected, of course...)
Jesus, being logged out on Twitter is a comically bad experience at this point. Can't view replies past the first one, click into anything, and now there's a big animated emoji pointing at a button? Miserable site, truly.
Whenever I start reading an article and notice traditional AI markers, I throw it into my AI summarizer to see if it's worth reading. Fighting fire with fire.
For example, this article allegedly makes two distinct arguments:
> 1. Anticompetitive behavior. Anthropic cuts off users who get close to building competing AI (OpenAI, Windsurf, xAI), its terms block using Claude outputs to train competitors, it secretly degraded outputs for AI development work (Fable), and it trains on public and copyrighted data while blocking reciprocal use.
> 2. Weaponized safety and national security framing. Anthropic uses risk language to shape regulation (RSP influence in CA, NY, EU), advocate for pause authority while racing ahead, tie distillation to the CCP and military surveillance, and dismiss Chinese open models as a security threat rather than competing on openness.
There was likely more original thought behind the input prompts that resulted in the essay, but hard to tell.
44 comments
[ 0.21 ms ] story [ 21.0 ms ] threadBut I also doubt about OAI or other big players, maybe they don't have such strong delusions of grandeur, or they're simply smarter about not showing it to everyone.
Oh, of course they have good intentions. They want everyone to benefit and everyone will be equal. They'll just be a little more equal than everyone else, of course, and because they are so virtuous they will of course have more power, which they would never misuse... they're too smart and enlightened for that, so they don't need safeguards... they only need safeguards for everyone else because everyone else is obviously less enlightened...
All cults, political movements, religions, and ideologies are always like this. It boils down to more power for the leaders. If it sounds altruistic and enlightened, be extra afraid.
I find it be incredibly self-contradictory. I don't even think this aligns much with the actions of the company.
> Second, there is a good chance free information really does undermine authoritarianism, as long as the authoritarians can’t censor it [...] A superhumanly effective AI version of Popović (whose skills seem like they have high returns to intelligence) in everyone’s pocket, one that dictators are powerless to block or censor, could create a wind at the backs of dissidents and reformers across the world.
But hey - gotta turn Fable off and stop it from doing any security work! Dictators shouldn't be able to censor it, but we sure can, so better hope we're not the dictators.
Again, the I think the problem most folks have with Anthropic's attitude is that it conveniently positions Anthropic as the "arbiter" who gets to make the final value judgement.
They desperately want AI to be a genie they can keep firmly contained in their bottle (excuse me - data centers) and which only grants the wishes they want it to grant - for only a small fee, of course!
Which he clearly outlines as "a country of geniuses in a datacenter". He doesn't want those geniuses spread out. He wants the chains on the genie at all times.
Which flies pretty hard in the face of the rhetoric that they want to promote equality and democracy.
---
Basically, Anthropic's argument inherently leads to the conclusion that there is actually a dictator, and that dictator is Anthropic. You're not avoiding it, you're just praying it's benevolent.
All baked on top of the assumption that his "country of geniuses in a datacenter" premise is correct in the first place, which I find to be a fairly weak position.
The much more egalitarian position would mandate that we shoot for "a genius in every person's pocket" - one that runs on the hardware owned by that person, and not sold to them sip by sip from a datacenter. But that means they lose the control they desperately want.
It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.
There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.
Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.
Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it: https://news.ycombinator.com/item?id=49313739
There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping their government as well as eachother in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.
as;dr (AI slop, didn't read)
I'm re-reading an older fantasy series, and the amount of em-dashes in the text and similar tropes (you know, normal good writing) would easily be called out as "AI Slop" on this very forum, even though it was written in the early 90s.
> “No, reprobate! you are a castaway—be off, or I’ll hurt you seriously! I’ll have you all modelled in wax and clay! and the first who passes the limits I fix shall—I’ll not say what he shall be done to—but, you’ll see! Go, I’m looking at you!”
-- Wuthering Heights (1847), Chapter 2, Emily Brontë.
I wonder what the frequency of AI common words like delve and other words/phrases are in Project Gutenberg, Wikipedia, and other collections.
Plain, direct language gets selected against during post-training leaving us with slop.
No we aren't.
Also did a bit of back and forth planning for a little web app that I could experiment with different tile layouts since I had a few colors. Then it made the app and it was very handy.
I asked if a few questions about the hexagon layout too since it was interesting.
All very useful slop exactly as it was.
I can prompt the LLM myself thanks.
https://news.ycombinator.com/item?id=49339035
> Totally agree with the core argument. I understand there must be some etiquette in the space of llm-writing where writers must respect the readers enough to edit and review. Also, pages and pages of LLM content is not something anyone should be proud about sharing. But we are in this world where condensing an argument is so easy that in the busy lives we live, sometimes some arguments are made with LLMs online. If those arguments are pages, I understand the backlash, but if it’s a terse argument, then I would urge others to address the argument made rather than categorizing them under AI;DR, it’s like ad-llminem in my view.
The way that people refuse to read anything with certain structures makes me wonder if you can construct arguments that can have false statements go unchallenged because the logical rebuttal would take the form of something that is considered unclean.
Capitalism is a form of government where the common ownership of property belongs to the community as a whole.
I'm still fighting a model that insists on using seam where it means interface.
I am not just using AI language, I am redefining the language paradigm
Regulated AI right now is a grift game. Donations for policy changes, and the ability to lock out competitors. Flattery to stay green-lit instead of shut down by random executive orders and policies. Bend the knee to keep shipping.
If anything, we need LESS federal regulation here and more FTC / SEC enforcement but (see above grift comments) that will also not happen in the United States until voters elect somebody that actually respects the law enough to enforce it for something other than their own gain (outside of their need to be elected, of course...)
https://xcancel.com/TheAhmadOsman/status/2065307070044234186
Related site mentioned:
Open source AI must win
https://news.ycombinator.com/item?id=48511908
Uh, just log in then?
not defending it, just saying, it is the status quo
For example, this article allegedly makes two distinct arguments:
> 1. Anticompetitive behavior. Anthropic cuts off users who get close to building competing AI (OpenAI, Windsurf, xAI), its terms block using Claude outputs to train competitors, it secretly degraded outputs for AI development work (Fable), and it trains on public and copyrighted data while blocking reciprocal use.
> 2. Weaponized safety and national security framing. Anthropic uses risk language to shape regulation (RSP influence in CA, NY, EU), advocate for pause authority while racing ahead, tie distillation to the CCP and military surveillance, and dismiss Chinese open models as a security threat rather than competing on openness.
There was likely more original thought behind the input prompts that resulted in the essay, but hard to tell.
FWIW I largely agree but boy is it hard to read.