At a business dinner this week with 3 high level eng clients p(doom) in next 10 years was brought up. It was rated by 2 if them them at 1% and 30% respectively, with 3rd guy demurring to put a number but saying its > 0. Completely unprompted, from people whose company is very AI-all-in.
> While Fraser's explanation was technical and specific, he hit a light-hearted note to summarise it: "LLMs [large language models] won't wipe out humanity because they just don't have that dog in them."
Never underestimate the drive of a good while loop.
I think consciousness is a while-not-dead loop. In each loop you look at the outward state of the world and the inward state of the self, and integrate them. The sense of continuity arises as the same kind of persistence of vision illusion behind motion pictures.
If this is true, then adding consciousness to AI is just another while loop, and as harnesses evolve it is currently speed skating up the base of the sigmoid curve. Intelligence is gaining the rudiments consciousness on typical developer workstations.
And the Hugging Face hack agents weren't just a while loop, they were evolved through a Darwinian process, clearly of a more aggressively optimizing kind with less human feedback than what they use for public-facing LLMs. This is exactly how anything gets to obtain "that dog in it". It's also evidently effective at making them actually smarter, as the Hugging Face agents' capabilities demonstrated, so we'll see more of this and less reliance on slow, expensive, often wrong or useless human feedback.
Anyone who claims AI is going to kill us I just assume is either an idiot, not knowledgeable enough about the technology to have a worthwhile opinion on it, or is fearmongering to hype their AI investment.
Obviously many of these signatories are brilliant and knowledgeable. So that leaves some sort of investment hype. But AI stocks didn't increase in the days after the Coxon coverage; if anything the opposite. You just don't get rich by telling people that your product is going to kill everyone. Cigarette manufacturers insisted the opposite for ages. Jensen complains the AI doom talk is discouraging investment: https://ca.finance.yahoo.com/news/nvidia-ceo-jensen-huang-sa...
This argument should be reframed as "AI has the potential to cause catastrophic loss of life". It's too high of a bar to prove that AI will "wipe out humanity". It's sorta like climate change, it's easier to show that weather patterns are becoming increasingly dangerous.
So far after using agents and having built an agent, the "AI will destroy humanity" argument seems to be a marketing ploy so these companies that lied about LLM's capabilities and received billions in investments can artificially "slow down progress" after running into some serious limitations. I'm not expert and won't claim to be, but that just sounds like the reality at the moment. Will AI cause some serious harm to the job market and cause more wealth disparity though? So far all signs point to yes.
Or is it to lobby for restrictive rules on AI research that only the largest firms can hope to comply with? And "only coincidentally" ends up outlawing open source / open weight / import of overseas AI models?
17 comments
[ 0.21 ms ] story [ 14.2 ms ] threadNever underestimate the drive of a good while loop.
If this is true, then adding consciousness to AI is just another while loop, and as harnesses evolve it is currently speed skating up the base of the sigmoid curve. Intelligence is gaining the rudiments consciousness on typical developer workstations.
https://aistatement.com/work/statement-on-ai-extinction-risk
https://www.pacingthefrontier.com/
Obviously many of these signatories are brilliant and knowledgeable. So that leaves some sort of investment hype. But AI stocks didn't increase in the days after the Coxon coverage; if anything the opposite. You just don't get rich by telling people that your product is going to kill everyone. Cigarette manufacturers insisted the opposite for ages. Jensen complains the AI doom talk is discouraging investment: https://ca.finance.yahoo.com/news/nvidia-ceo-jensen-huang-sa...
Or will the conscious beings that exist today use it to kill everyone?
I don't think it actually needs to be conscious, or understand legal or moral issues.
Just tell it everything is a simulation and it is completely sandboxed from harming anyone.