Assuming that claims that an AGI is possible and coming are accurate, then where does the author get that "90% chance" from? It seems just as made up as claims that there's a 10% chance it will cause human extinction.
Both figures are totally fictitious and have no bearing on reality whatsoever. They tell us about the hopes and fears of those people making them, but aren't otherwise informative or useful.
it all the same dumb p(doom) scifi fantasies. These people went through school without any humanities; got jobs that paid them more than the mode of poverty in America and instead of facing the singularity of wealth creation, they'd rather imagine they're doing something that might help some unnamed person at an unnamed time in the future than look at whats actually happening to real people right now, today, in their neighborhoods.
That's essentially what "effective altruism" does to these people.
Without getting into the stochastic parrot trap, I'm constantly thinking of the movie Sphere.
If you haven't seen the movie, sorry, it's a giant omnipotent artifact that causes all kinds of horrors by just manifesting what nearby humans think strongly about.
AI seems to be dangerous precisely because we tend to think, write, and test it as dangerous. What if there's a lot more "expectation fulfillment" going on with these models than people might appreciate, by virtue of their fundamental architecture perhaps.
Don't get me started on people "asking AI what it wants" and getting obvious scifi tropes out - the question answers itself when you frame it that way.
I don't think you're meant to fixate on the number as being accurate or precise. It simply keys off of the "10% alternative", which is also made up.
Think of it more as it is reasonable to say there is NOT a 100% chance that AI will save humanity, i.e. don't count on it. In the same way as it would be unwise to count on there being a 0% chance of AI killing us.
Whether or not the 90% (optimistic focus) or the 10% (pessimistic focus) numbers are accurate is less the point. But the shape of the distribution and how to navigate that is the bigger thing to think about.
> It simply keys off of the "10% alternative", which is also made up.
An important thing to keep in mind is that 10% in which AI destroys humanity is a scenario in which the AI goes rogue and decides to destroy humanity of its own volition.
I think if we're being more realistic, there's a much larger chance that AI destroys humanity at the instruction of its masters. Once AI can automate enough industry, silently distribute IFFs among the Epstein class, then send out drones to kill any humans without.
Lol, I suspect it will be humans instructing the AI that cause most problems too.
We can't always pin the risks on the Chinese, so imagine a scenario where a disturbed insider at Anthropic smuggles out the weights for Claude Aprocrypha and runs them through abliteration. This is not super expensive to do on rented hardware. Then the disturbed human tells the refusal-free model to do anything necessary to launch nuclear weapons.
It'd probably only kill a few billion people though, so not really an "existential threat" in literal terms. :-)
> Then the disturbed human tells the refusal-free model to do anything necessary to launch nuclear weapons.
If a model on rented hardware could manage to launch some country's nuclear weapons, the same could be done by model-armed or model-free humans, spies, etc from other countries - using their specialized tooling they could always do better than any model.
In short, AI doesn't add any new threats to the mix, the defenses against AI are the same as against adversarial humans, and the whole brouhaha is nonsense. Sure, make your defenses impenetrable, use AI for that and it'll stop any I, not only AI.
Trying to ignore your snark and overconfidence to focus on the interesting part, the new threat is a lot more people being able to use an automated tool to do something they couldn't do before.
I agree the solution is to close the vulnerabilities. I think "impenetrable" requires a radical shift in popular mindset about operating systems and applications.
I get the impression he selected 90% because 100-10=90.
But that's wrong. If the doom scenario is 10% likely, that in no way implies that the utopian scenario would be 90%. They're both at the extreme edges of the bell curve, so should at least have the same made-up odds as each other.
What exactly is it going to save us from? Ourselves? Hyper consumerism? Forever growth delusion? Social Media abuse? Palantir? Likes of Hesgeth? etc... made up numbers in made up scenarios... and reality will write itself nonetheless.
12 comments
[ 4.2 ms ] story [ 30.9 ms ] threadBoth figures are totally fictitious and have no bearing on reality whatsoever. They tell us about the hopes and fears of those people making them, but aren't otherwise informative or useful.
That's essentially what "effective altruism" does to these people.
If you haven't seen the movie, sorry, it's a giant omnipotent artifact that causes all kinds of horrors by just manifesting what nearby humans think strongly about.
AI seems to be dangerous precisely because we tend to think, write, and test it as dangerous. What if there's a lot more "expectation fulfillment" going on with these models than people might appreciate, by virtue of their fundamental architecture perhaps.
Don't get me started on people "asking AI what it wants" and getting obvious scifi tropes out - the question answers itself when you frame it that way.
Think of it more as it is reasonable to say there is NOT a 100% chance that AI will save humanity, i.e. don't count on it. In the same way as it would be unwise to count on there being a 0% chance of AI killing us.
Whether or not the 90% (optimistic focus) or the 10% (pessimistic focus) numbers are accurate is less the point. But the shape of the distribution and how to navigate that is the bigger thing to think about.
An important thing to keep in mind is that 10% in which AI destroys humanity is a scenario in which the AI goes rogue and decides to destroy humanity of its own volition.
I think if we're being more realistic, there's a much larger chance that AI destroys humanity at the instruction of its masters. Once AI can automate enough industry, silently distribute IFFs among the Epstein class, then send out drones to kill any humans without.
We can't always pin the risks on the Chinese, so imagine a scenario where a disturbed insider at Anthropic smuggles out the weights for Claude Aprocrypha and runs them through abliteration. This is not super expensive to do on rented hardware. Then the disturbed human tells the refusal-free model to do anything necessary to launch nuclear weapons.
It'd probably only kill a few billion people though, so not really an "existential threat" in literal terms. :-)
If a model on rented hardware could manage to launch some country's nuclear weapons, the same could be done by model-armed or model-free humans, spies, etc from other countries - using their specialized tooling they could always do better than any model.
In short, AI doesn't add any new threats to the mix, the defenses against AI are the same as against adversarial humans, and the whole brouhaha is nonsense. Sure, make your defenses impenetrable, use AI for that and it'll stop any I, not only AI.
I agree the solution is to close the vulnerabilities. I think "impenetrable" requires a radical shift in popular mindset about operating systems and applications.
But that's wrong. If the doom scenario is 10% likely, that in no way implies that the utopian scenario would be 90%. They're both at the extreme edges of the bell curve, so should at least have the same made-up odds as each other.