- "The journal Science even published a piece on the question, in June 1946, on the eve of the US's post-war tests in Bikini Atoll. It stated that not only "non-scientists" are "disturbed over the prospect" of planetary ignition."
Huh: that linked Science piece is actually not correct. Its safety argument is that nuclear fission of silicon and oxygen, the most common elements in the Earth's crust, are endothermic. That's true. It's also backwards: the actual problem is that nuclear fusion of light elements is exothermic, and in fact self-sustaining under certain conditions (extreme mass density).
The piece doesn't touch nuclear fusion at all. Was the expert not aware of it? He should be: he's discussing (astrophysical) novae as an analogy – and those are precisely runaway thermonuclear fusion detonations. (Didn't they know that in 1946?)
To be fair, he does clearly disclaim in that piece that he isn't a nuclear physicist. Still: it's remarkable that it's totally wrong, given the severity of the question it's attempting to answer (and the authoritative platform it's published in).
> Though analogies between artificial intelligence and nuclear weapons can be overstretched, the way this concern went from idle speculation to serious concern can teach us lessons regarding fears swirling around today's emerging technologies.
Sounds like an overstretched analogy.
There is no such principle which says that just because a theoretical concern in one area turned out to not be a problem in practice, all theoretical concerns will not be a problem in practice.
I think the analogy is apt. Same way the atmosphere is not a bag of nuclear gunpowder, our crappy, slow data centers (and even sand on the beaches that could be turned into silicon) are nothing even close to something being capable of powering humanity devastating AI singularity.
It's not algorithms that stand in the way, it's crap computational capacity of our local physical reality.
And people thought one nuke might start chain reaction of Earth atmosphere. It turned out it's not possible.
Single human runs on 100W with algo and hardware optimized for more than million years. I think at some point we can do better with sand and electricity, but it's still far away.
And AI research is terrible with algos. Modern miracle GPT is plain old multilayer neural network with some forward shortcuts and some parts hard coded according to ad-hoc "attention" idea. And learning algorithm is more than half a century old. The reason GPT performs is just the RAM and compute.
We have every reason to be sceptical about AI apocalypse.
When it comes to safety you play on the safe side, not the dangerous one. I don't share the philosophy of amateur submariners and I've had my fill of their hollow and reckless arguments. A handful of humans (a few hundred watts of compute) succeeded at devastating the world from 1939-1945. AI already knows more domains of knowledge, is faster, and does not require 20 years to spin up each new instance.
Just because 2 of the 10 chains restraining the cave troll are still intact does not mean it is time to make tea.
> I don't share the philosophy of amateur submariners
That makes perfect sense in 2023. Not so much in seventeen hundreds, or whenever people first started to attempting it.
> A handful of humans (a few hundred watts of compute) succeeded at devastating the world from 1939-1945.
I wouldn't really count it towards how powerful a person is. Literally billions of people amount to almost nothing their whole life. Wars were results of historical processes. People are nothing than a match under the barrel of gunpowder filled with economical and societal forces.
If there are such forces that could end us what's the difference if the match will be biological or silicon?
> Just because 2 of the 10 chains restraining the cave troll are still intact does not mean it is time to make tea.
16 comments
[ 2.9 ms ] story [ 99.5 ms ] threadHuh: that linked Science piece is actually not correct. Its safety argument is that nuclear fission of silicon and oxygen, the most common elements in the Earth's crust, are endothermic. That's true. It's also backwards: the actual problem is that nuclear fusion of light elements is exothermic, and in fact self-sustaining under certain conditions (extreme mass density).
https://www.science.org/doi/10.1126/science.103.2687.760.a
The piece doesn't touch nuclear fusion at all. Was the expert not aware of it? He should be: he's discussing (astrophysical) novae as an analogy – and those are precisely runaway thermonuclear fusion detonations. (Didn't they know that in 1946?)
To be fair, he does clearly disclaim in that piece that he isn't a nuclear physicist. Still: it's remarkable that it's totally wrong, given the severity of the question it's attempting to answer (and the authoritative platform it's published in).
Sounds like an overstretched analogy.
There is no such principle which says that just because a theoretical concern in one area turned out to not be a problem in practice, all theoretical concerns will not be a problem in practice.
It's not algorithms that stand in the way, it's crap computational capacity of our local physical reality.
Single human runs on 100W with algo and hardware optimized for more than million years. I think at some point we can do better with sand and electricity, but it's still far away.
And AI research is terrible with algos. Modern miracle GPT is plain old multilayer neural network with some forward shortcuts and some parts hard coded according to ad-hoc "attention" idea. And learning algorithm is more than half a century old. The reason GPT performs is just the RAM and compute.
We have every reason to be sceptical about AI apocalypse.
Just because 2 of the 10 chains restraining the cave troll are still intact does not mean it is time to make tea.
That makes perfect sense in 2023. Not so much in seventeen hundreds, or whenever people first started to attempting it.
> A handful of humans (a few hundred watts of compute) succeeded at devastating the world from 1939-1945.
I wouldn't really count it towards how powerful a person is. Literally billions of people amount to almost nothing their whole life. Wars were results of historical processes. People are nothing than a match under the barrel of gunpowder filled with economical and societal forces.
If there are such forces that could end us what's the difference if the match will be biological or silicon?
> Just because 2 of the 10 chains restraining the cave troll are still intact does not mean it is time to make tea.
Imagined cave troll.
I know it's not the same because AI research is closer to social sciences than to physics but still, there are parallels.