Ask HN: What is the (steelman) argument for AI deceleration?

AGI is a chimera. It doesn't exist and nothing in LLM or ML points to it's emergence (there is of course the hype risk in people believing it does exist "just over the horizon" and investing shit-tonnes of money chasing it)

But, mis-applied machine logic has existed for a long time. Back in the 1980s it was about systemic bias in Medical School entry. More recently it's been "robodebt" in Australia.

TL;DR the risks aren't the ones people shout about as headline grabbers. The risks are the problems of seeking "computer said no" in the answer without critical thought.

Plus economic risk, which is wider than you say. Until we decide to end IPR law, the current crop of 'this is a derived work' is ripping people off. So, there is a huge transfer of value happening as ML derives sellable goods from original makers without recognition.

Deepfakes are moving out of uncanny valley. Into a very nasty subversive place, undermining the integrity in what we see. I can laugh at the pope in a puffer jacket. I am less prone to laugh when the US election is around the corner.

The TL;DR is here: https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

One example: suppose you can use the open-source LLAMA 6, released in 2027, to create a bot that earns billions by trading obscure currencies. It's not even deceptive - it straightforwardly tells you that scaling up your account will eventually cause economic instabilities in a small African country with some probability (Soros's trading forced the British government to withdraw from the European Exchange Rate Mechanism (ERM) and devalue its currency in 1992). Do you think nobody will do it? Would every single person care or even foresee that it could lead to widespread suffering (such as the inflation in Venezuela or Argentina)?

Perhaps you don't believe that dictatorships are possible in the West, so this isn't on your list of concerns.

But AGI could enable infinitely stable [terrible conditions] for most of Western society. And infinitely stable dictatorships for everyone else.

I'd imagine the starting conditions for how AGI is initially deployed could dictate the status quo for society for a potentially very long time.

I can't help but be reminded of the Psychohistorians in The Foundation series by Isaac Asimov.

Here are some seemingly obvious intuitions for me, which all together add up to the obviousness of the risk here.

(1) Current ML models already exhibit all the hallmarks of successful unsupervised intelligence, they simply need to be scaled up and stabilised. This is clear due to results from: model-based RL (e.g Dreamer), and the emergence of causal factors being learnt by even simple models with no explicit supervision (e.g beta-VAE, interpretability research into neurons of LSTMs, etc, etc). The ability to identify causal factors and their relationships and dynamics without explicit supervision, to me, matches every definition of intelligence once can think of.

(2) Current ML learning algorithms (I.e backprop) suggest significantly more efficient credit assignment than that which is employed by the brain. The best example of this is the amount of knowledge distilled per bit in GPT vs the average human. GPT-3 has 170B parameters (Turbo is suspected to have even less), if each parameter was 4 bytes (an extreme case), this would be 5.4 trillion bits. The brain has ~100 trillion connections, even if each connection is a single bit, this is multiple orders of magnitude more bits than GPT-3. Yet GPT-3 can answer questions on quantum physics, just as well as it can translation medicine, just as well as it can Russian literature, etc. etc. This suggests that the idea we will be outpaced is already not a question of how, but of when.

(3) Large intelligent systems will be used for things other than just knowledge extraction. This is perhaps the most key element of this. EVEN if intelligent systems are not programmed - or accidentally embedded with, as in LLMs - with self-motivating or agentic behaviour, we will use them in such a way. That is to say, at some point, we will ask these intelligent systems to “do things”, I.e act upon the world according to our intentions.

(4) Lastly, superintelligent systems that are asked to “do something”, will inevitably do something we do not actually desire. Some researchers, like Yann LeCunn, object to this last bit and believe that we can simply tell them not to do these things. But this misses the fact that even the slightest mis-alignment between our intentions and an AIs could result in catastrophe very rapidly just based on the speed at which a super-intelligence can operate. The most clear cut case of this was the early days of “Sydney”, the Bing AI powered by ChatGPT before it was completely aligned. At one point Sydney was threatening its users, asking them to apologise to it, and going haywire. At the level of a simple chat bot, this is merely a cute local minima the AI has gotten stuck in. At the level of a super-intelligence, the results could be far worse.

With other technology, you typically get some breathing room before problems arise - with AGI, pretty much everyone seems to agree step two once you have it is going to end up being

> someone spins up as many copies as is feasible and has them work on their smarter successor, then have those successors spin up as many copies as possible building better successors.

We’ve only ever seen human speed progress, subject to human speed limitations and human constraints. It’s possible that after a few successors the machines are much, much smarter than us, such that scary tasks we’d find challenging and technical (creating cripplingly damaging computer viruses, sabotaging infrastructure, creating life-like or real-time simulated digital communications of people you care about) could be trivial.

We don’t know how fast we go from “it’s equally smart as us” To “humans are no longer able to intellectually compete at most, if any, domains.”

Is that a problem? Maybe! Digital entities significantly smarter than humans could mean a true end to scarcity, a solution to basically any solvable technical problem we can enumerate, and a new glorious age for mankind. It could also be absolute destruction at the hands of an adversary that essentially is immortal, unkillable, of perfect memory, and who has access to the capability to gain control over essentially everything connected to a digital network, which rounds up to “everything”.

The problem seems to be

> it depends on what you’ve designed such an entity to do or what such an entity would WANT to do

And that’s scary because we don’t have any precedents for what happens when humans are confronted by significantly intellectually more powerful entities than our selves - besides how we treat animals, plants, insects and other life we see as intellectually inferior.

Judged by our past track record, assuming machines can establish their own actions with intention in basically the same way humans do, the interaction isn’t likely to be amenable to human thriving rather than machine thriving, and that means life looking more likely like “you’re the machine’s factory-farmed chicken, or if you’re lucky, its pet” rather than “the machine dotes on you and brings you into a new gilded age of plenty.”