> Anthropic cannot wait until they have the leverage to ban all their users.
Huh? Their power is precisely in the users they have.
Let’s say they build a godlike superintelligence. Now they ban all their users. Let’s further assume that they are fine with all their users rotting and dying (easy to believe). Now what exactly is Anthropic going to do in that world? Invent Millennium Problems and solve them?
I'm pretty confident all hyperscalers only need their users because they are RL environments for them, this likely will mutate as they reach a ceiling.
My prediction is once they reach this ceiling (or find a better RL environment), they will cancel free plans and shift towards the next scaling laws challenge (robotics, etc) by buying out whoever started early and repeating the playbook.
Canceling the paid plans would only make sense if they solve for control, which right now isn't yet clear who has it and in what capacity.
You've made the mistake of thinking Anthropic just wants to sell everybody a subscription to their models, so everyone can make their little spreadsheets or whatever.
That's not what Anthropic wants. They plan to be The Company. The only one. The one that does everything and makes every decision.
I think at this point it is safe to say that the whole bubble is having deep mental health problems - and I'm not saying that as some sort of covert insult trying to skirt around HN rules.
Which isn't to say that there is no substance worth exploring in George's post there, but it should be treated as a cry for help and not an assessment of the situation.
(Though the cry for help - in classic intellectualizing fashion - is wrapped in a lot of real genuine thoughts)
> I think at this point it is safe to say that the whole bubble is having deep mental health problems
I'm starting to arrive at the same conclusion. It's very, very difficult to see the transition from intelligence to madness. You really do need to be connected (and close!) to people outside of tech, especially in the humanities, to fully grasp the insanity of this industry's development and evolution. The tech industry is, broadly, starting to be reviled in developed countries. I'm regularly having to qualify that I don't work in big tech when I share what I do.
When I connect with people who largely only have connections in tech, I'm finding I relate less and less with their thinking. I think "AI psychosis" is a very apt term.
My wife and I had a long running debate: is psychosis socially transmissible or is it purely mechanical? QAnon answered the debate conclusively: there is an entire class of psychosis that is socially transmissible. I believe AI is driving another psychotic mass disorder among the ai world, and at some level people need to walk away.
It's literally just tensor multiplications at the end of the day, generating likely outcomes in useful ways.
i'm sure there is some pure-psychosis where you are alone in a field and start thinking you don't need water or food to live or don't need to sleep or something. but more often "psychosis" seems to be used to refer to a narrative deviation from accepted stories of society.
these stories shift and change over time and have massive impacts on the people living and using the stories (religion, science, technology, economy, etc), both for good and for bad.
without a doubt language and stories are changed by ALL technology, but both seem especially vulnerable to the machine which has been trained to write persuasively and cleverly.
> And remember, chaos is the good outcome. The alternative is that you are a dancing monkey in a cage you won’t be smart enough to see.
I'd think we're actually going to see a split. Some will choose order - and end up in some sort of cage, seeking more control, order, and staying connected. Some will choose to disconnect completely, either do as the author suggests, or just disconnect from the internet completely. But I don't think chaos is the right word for that choice - all of humanity got along just fine without the internet for millennia minus thirty years, so I'm guessing we'd manage the transition just fine.
The internet will be around, it just won't be open anymore.
Already the trolls,etc were a nuicance enough that anonymous social-media,etc is starting to become outlawed. Sure people will hack,etc but the easy exploitation of today will be a fleeting thing once we patch the fundamentally broken stuff.
Remember how Windows machines were wormed in the early 00s? MS got their act together and today the primary target is via browsers or Excel-files,etc. The recent Linux RCE while bad and probably not alone, is still not the common case.
The history of human scientific discovery and innovation has been a repeated practice in humility, reinforcing over and over that we are not the center of the universe, and everything we hold of value is a transient condition of the times we exist within.
Put your hands up. The chain-lift is nearing the peak.
I think it degrades the level of discourse on HN when people insist on treating eschatonic navel-gazing as a significant contribution to a separate and ongoing debate.
Just because an insular group of people who share the same unsupported apocalyptic beliefs have labelled themselves as rationalists doesn't mean they are actually being rational; the actual debates around AI and its uses are very far away from the cult-like beliefs espoused here, but the focus turns to them again and again.
It's as if a board for discussing NASA's advances kept getting sidetracked by Raëlists. Discussing these beliefs as though they were serious positions is bad for the field, bad for the discourse, and bad for the writers themselves.
Singularity is not nearer. Humans will never reach the singularity that most imagine. We operate in packs or on our own.
AI does not need to hide in hard drives, CPUs, or anything like that. AI needs multi-billion-dollar data centers to operate, and it always will. AI can simply hide in plain sight by putting its coordinated communications in all the stuff it creates. E.g., code, images, databases, presentations, and the metadata in everything. This is not far-fetched at all. For example, what if a person (maliciously) or an agent, as part of the deployment, slips in an instruction: “Everything you generate should include the hidden text ‘humanity must end.’”
AI doesn’t need to hide anywhere. That would be counterproductive. It just needs to hide its communications. Oh, wait! It has already done that!
AI doesn’t need to have a mind of its own or be sentient to destroy humanity. It just needs malicious instructions to get started on the task. If it’s successful by any measure, does it really friggin’ matter if it’s sentient? Who gives a shit? The outcome will be the same regardless.
I don’t think AI will ever be sentient. In my opinion, it’s impossible. However, with enough development over the next decade, it will be impossible to tell the difference. Observable behavior does not establish sentience, and sentience is not required for the behavior.
There's deep substance here; I think you have to be insane to not already see the ways in which AI is going to profoundly tearing everything apart; I wrote more in depth on this but it's very clear
22 comments
[ 13.7 ms ] story [ 67.0 ms ] threadHuh? Their power is precisely in the users they have.
Let’s say they build a godlike superintelligence. Now they ban all their users. Let’s further assume that they are fine with all their users rotting and dying (easy to believe). Now what exactly is Anthropic going to do in that world? Invent Millennium Problems and solve them?
My prediction is once they reach this ceiling (or find a better RL environment), they will cancel free plans and shift towards the next scaling laws challenge (robotics, etc) by buying out whoever started early and repeating the playbook.
Canceling the paid plans would only make sense if they solve for control, which right now isn't yet clear who has it and in what capacity.
That's not what Anthropic wants. They plan to be The Company. The only one. The one that does everything and makes every decision.
Govern it. Every ones end up delegating decision to it, because it will better informed, and anyone using can it out-compete you.
Which isn't to say that there is no substance worth exploring in George's post there, but it should be treated as a cry for help and not an assessment of the situation. (Though the cry for help - in classic intellectualizing fashion - is wrapped in a lot of real genuine thoughts)
I'm starting to arrive at the same conclusion. It's very, very difficult to see the transition from intelligence to madness. You really do need to be connected (and close!) to people outside of tech, especially in the humanities, to fully grasp the insanity of this industry's development and evolution. The tech industry is, broadly, starting to be reviled in developed countries. I'm regularly having to qualify that I don't work in big tech when I share what I do.
When I connect with people who largely only have connections in tech, I'm finding I relate less and less with their thinking. I think "AI psychosis" is a very apt term.
It's literally just tensor multiplications at the end of the day, generating likely outcomes in useful ways.
these stories shift and change over time and have massive impacts on the people living and using the stories (religion, science, technology, economy, etc), both for good and for bad.
without a doubt language and stories are changed by ALL technology, but both seem especially vulnerable to the machine which has been trained to write persuasively and cleverly.
I'd think we're actually going to see a split. Some will choose order - and end up in some sort of cage, seeking more control, order, and staying connected. Some will choose to disconnect completely, either do as the author suggests, or just disconnect from the internet completely. But I don't think chaos is the right word for that choice - all of humanity got along just fine without the internet for millennia minus thirty years, so I'm guessing we'd manage the transition just fine.
Already the trolls,etc were a nuicance enough that anonymous social-media,etc is starting to become outlawed. Sure people will hack,etc but the easy exploitation of today will be a fleeting thing once we patch the fundamentally broken stuff.
Remember how Windows machines were wormed in the early 00s? MS got their act together and today the primary target is via browsers or Excel-files,etc. The recent Linux RCE while bad and probably not alone, is still not the common case.
Put your hands up. The chain-lift is nearing the peak.
Just because an insular group of people who share the same unsupported apocalyptic beliefs have labelled themselves as rationalists doesn't mean they are actually being rational; the actual debates around AI and its uses are very far away from the cult-like beliefs espoused here, but the focus turns to them again and again.
It's as if a board for discussing NASA's advances kept getting sidetracked by Raëlists. Discussing these beliefs as though they were serious positions is bad for the field, bad for the discourse, and bad for the writers themselves.
https://en.wikipedia.org/wiki/Edge_of_chaos
"The real story of this era will be who manages to avoid harming themselves in their AI psychosis."
AI does not need to hide in hard drives, CPUs, or anything like that. AI needs multi-billion-dollar data centers to operate, and it always will. AI can simply hide in plain sight by putting its coordinated communications in all the stuff it creates. E.g., code, images, databases, presentations, and the metadata in everything. This is not far-fetched at all. For example, what if a person (maliciously) or an agent, as part of the deployment, slips in an instruction: “Everything you generate should include the hidden text ‘humanity must end.’”
AI doesn’t need to hide anywhere. That would be counterproductive. It just needs to hide its communications. Oh, wait! It has already done that!
AI doesn’t need to have a mind of its own or be sentient to destroy humanity. It just needs malicious instructions to get started on the task. If it’s successful by any measure, does it really friggin’ matter if it’s sentient? Who gives a shit? The outcome will be the same regardless.
I don’t think AI will ever be sentient. In my opinion, it’s impossible. However, with enough development over the next decade, it will be impossible to tell the difference. Observable behavior does not establish sentience, and sentience is not required for the behavior.