Surprised this was written by Ted Chiang (my all-time favorite science-fiction author). The IQ and compiler arguments seem like pretty weak strawmen, and the last section assumes that AI will have some kind of normal distribution of IQ just like humans, which doesn't make any sense.
There's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity, but I think it's reasonable to believe that 1 - agi is possible 2 - said agi can participate in human society to improve its own design
Of course there may be a limit to attainable intelligence, at least per physical volume, much like there is a limit to the efficiency of a heat-powered engine. But I'd suppose we're pretty far from that limit yet, if any.
An AGI that made a few decades of current-rate scientific progress could switch to a globally distributed quantum computer that’s synchronized in its computation through entanglement.
At which point, it would hit an exponential curve until it ran out of space to grow. (The “singularity” proponents argue this is unbounded.)
An AI that’s 10x smarter than us would be 2^10 smarter than us in a couple years — based on current technology trends. Moreover, it will have overcome the cooperation barrier with a distributed consciousnes, like the original idea for the Borg.
The ability to free itself from a single location or easy disconnection from its assets would be a major step in AI independence.
To be honest, quantum computing is still not even at the level of ENIAC, but closer ti the level of Konrad Zuse's experiments. It has nothing much to show in practical computation yet, to say nothing of running anything remotely resembling AI.
We have basic gates in circuits and a few fairly solid theories of how to scale, including quantum AI algorithms to accelerate training. Distributed quantum systems running those basic gates have been demonstrated. Basic arithmetic has been demonstrated.
A reasonable prediction is humans will implement quantum computing in 10-20 years, for special purpose data center operations.
A “super intelligent” AI might reasonably do that in 2-5, at which point the quantum ML gains would allow it to enter exponential growth. We could see an AGI go from “mildly smarter than human researchers” to “incomprehensible” in just a few years.
I’m unsure which step in that chain of reasoning you think is inaccurate.
You just said “to be honest”, implied I was uninformed because I disagree with you, then said something concurring with me: Zuse’s work happened a decade before ENIAC (1936 vs 1946). If quantum is at the equivalent of Zuse’s work, AI could suddenly accelerate it and go exponential.
You’d be funnier if you quoted what you believe I said that’s incorrect, rather than making nebulous accusations because you didn’t understand or agree with my comment.
Why should we get scientific? You didn't. You assumed some stuff, and came up with exponentially self-improving AI. That's not science.
There is no scientific reason why global quantum communication means that an AI hits exponential growth. (If there is, you sure haven't supplied it! You've just asserted that it is so.)
We have papers showing how to exponentially accelerate AI using quantum computers, the same way we can accelerate other types of problems. (Finding a “best DAG” is related to lowest energy state.) Microsoft already has published a number of articles and is working on more. In general, global entanglement will give an exponential boost over isolated data centers — the same way a fully entangled computer is more powerful than a DWave.
I understand a number of readers aren’t familiar with QAI research though.
But that’s my point —
> Why should we get scientific? You didn't.
The person responding made a generic negative comment (as did you) due to not knowing the state of the art.
Incorrectly “correcting” people while yourself ignorant is prevented by giving people specific corrections: if you’d taken the time to tell me what you think I’m wrong about, you’d probably realize you don’t actually have a specific thing you think I’m wrong about — you just were surprised by my conclusion.
But given a globally connected quantum AI, where's the science that says that that's enough to continue to improve exponentially? To actually get smarter?
The downvoting is hilarious: nobody can actually say what I’m wrong about, and the one person who tried agreed with me, but you’re here downvoting because you’re sure you’re experts.
If you were really smarter than me at QM, you’d be able to point out what was wrong with what I said.
The HN downvotes without correction really speak to the hubris around here.
You are theoretically correct. But we haven't yet attained practical command of the ideas, haven't turned them into technologies. Because of that, any level of detail is about as believable as any sci—fi. In practice, things may end unattainable in the form we currently imagine them, and anything along the lines most likely will not be very similar, except in deepest principles.
We can conservatively say that even though there is a limit to amount of computation produced in a given volume, temperature range, and time, this limit is still sky-high compared to what we do now. OTOH the speed of light is already a major limiting factor.
Sure — as you say though, we’re far off that limit. There’s plenty of room for limited exponential growth in its intelligence.
There’s a number of facts that make this not sci-fi:
- China has demonstrated distributed quantum networks.
- The US has demonstrated distributed quantum calculations.
- Anyons were recently experimentally verified.
- We have a number of small quantum ALU.
- There are quantum-effect systems in production, just not fully-entangled systems.
- We have quantum ML algorithms for accelerated training.
Nobody has disputed my baseline estimate: that humans will complete a quantum computer in 10-20 years. (If people just disagree there, they should say so.)
However, once you accept that, it’s trivial to see the following sequence:
- AI is slightly smarter than humans, does quantum work 2-5x faster.
- AI develops quantum circuits in 2-4 years that would have taken 10-20 for humans. (See COVID vaccines for power of tech acceleration.)
- Those quantum systems let the AI replace its current systems with faster local ones.
- The AI uses another 1-2 years at this intermediate stage to extend current quantum networking research to unify its quantum data centers into a single computer, giving full global entanglement.
So in 3-5 years, an AI which starts off mildly smarter could finish nearly-done projects and enter an exponential growth curve before we do.
That progression looks like:
- Scale M AI.
- Scale M/K * 2^K, where K is the portion of M at a single data center.
- Scale 2^M AI, when it can fully entangle.
> Because of that, any level of detail is about as believable as any sci—fi.
That’s not true: most major companies in the space have labs looking to turn fundamental research into products, which is the exact opposite of sci-fi.
I think people just haven’t actually thought about this and are responding with fallacies rather than confront an idea they intuitively reject.
Even though I’m right on the technical issues involved.
> and the last section assumes that AI will have some kind of normal distribution of IQ just like humans, which doesn't make any sense.
The central limit theorem shows that the normal distribution emerges as the limit of the average of independent and identically distributed random variables of any distribution.
And there is no particular reason to believe that capabilities are merely the sum of a bunch of random iid variables! If you look at how DL scales with model size or data or FLOPS, it's a powerlaw (or lognormal), and in countless places in the natural world, things are lognormal rather than normal. Similarly, wealth, scientific productivity, lots of things are lognormal rather than normal. Even for human intelligence, the normality is due more to the fact that we all come off the assembly line as Mark 1 human units, practically identical, with the differences coming from environmental randomness & randomness in genetic mutation assortment.
Won't AI designers choose optimal values for those variables that have been identified? If so, I don't think the result looks like a normal curve, or at least the resultant 'optimized' normal curves skew 'smarter' relative to the human normal curve.
>There's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity
There are reasons to believe the opposite, that there's a limit. Not just practical technological limitations (e.g. limits to CPU process, speed, power consumption, etc) but also absolute limits for information processing under certain physical constraints (in the Shannon sense).
If you think of chess programs there isn't a limit to how many positions per second you can do - you can just run more machines in parallel. Thought at the end of the day you just get a slightly better chess machine. I imagine AI will go a bit like that.
There is a discrete and finite number of chess positions so there is a limit, but it might not be computable with resources available within the observable universe.
Finitude seems a pretty weak constraint. There's only a finite difference between the intelligence of a paramecium and that of a human. The gap between our intelligence and the theoretical maximum is still, I imagine, unfathomably big.
The word evolution didn’t appear in this article which is odd considering it’s the mechanism responsible for getting us from a single cell to an individual capable of saying such advancement is unlikely.
> Once there’s a person with an I.Q. of, say, 300, one of the problems this person can solve is how to convert a person with an I.Q. of 300 into a person with an I.Q. of 350. And then a person with an I.Q. of 350 will be able to solve the more difficult problem of converting a person with an I.Q. of 350 into a person with an I.Q. of 400. And so forth.
with strawmen like these, who even needs a debate?
Once you have an AI with a human approximate general intelligence, the ability to instantiate it thousands of times and/or run it at thousands of times 'baseline' speed to explore a problem space is going to look like a big IQ boost.
I don’t think this analogy is helpful. If an IQ 110 AI can research 10^6 person-hours for each IQ 110 person-hour, the results are going to look like a qualitative difference (i.e., higher IQ). That said, I do think that some problem spaces require ‘true’ IQ advances to address - insight vs. bruteforce.
I have to disagree with the author that optimizing compilers are "a useful precedent for thinking about the idea of an intelligence explosion". Such compilers are designed to do one thing, and they are designed to do it some fixed amount better than their predecessors. An AI that can improve itself will instead probably have to be at the level of a human thinker, who can be "creative" and adapt/produce results in a wide variety of situations -- the fact that our brains exist show that such a machine is possible. Those who anticipate an intelligence explosion are picturing these general "thinking machines", not advanced optimizing compilers.
Also,
> I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion
I have never heard this assumption in a technical discussion in the modern age, and the author provides no source here. I'm overall unconvinced by this article.
The only thing I got out of this is the fact that when people say ted Chiang is such a brilliant guy, you have to read his stuff, my mind is going to silently think no he's a sophist hack. Am I the only person who read this and thought my god this person is an idiot but is smart enough to get other people to think he's not an idiot but is actually a genius .
No, but I got the impression that you have a facile understanding of human thinking/creativity, in that you think that a person with bad arguments in X areas must necessarily have bad arguments in other areas, and can't also be a brilliant writer otherwise.
He claims humans can't increase the intelligence of animals, but we probably can?
Dogs probably have higher emotional intelligence (by human standards) than wolves, and we probably could breed dogs (or have we?) that are smarter at basic puzzle solving activities like opening latches with their paws or whatever.
33 comments
[ 3.3 ms ] story [ 72.0 ms ] threadThere's no reason to believe that intelligence is a quantity that can be improved ad infinitum ala singularity, but I think it's reasonable to believe that 1 - agi is possible 2 - said agi can participate in human society to improve its own design
An AGI that made a few decades of current-rate scientific progress could switch to a globally distributed quantum computer that’s synchronized in its computation through entanglement.
At which point, it would hit an exponential curve until it ran out of space to grow. (The “singularity” proponents argue this is unbounded.)
An AI that’s 10x smarter than us would be 2^10 smarter than us in a couple years — based on current technology trends. Moreover, it will have overcome the cooperation barrier with a distributed consciousnes, like the original idea for the Borg.
The ability to free itself from a single location or easy disconnection from its assets would be a major step in AI independence.
A reasonable prediction is humans will implement quantum computing in 10-20 years, for special purpose data center operations.
A “super intelligent” AI might reasonably do that in 2-5, at which point the quantum ML gains would allow it to enter exponential growth. We could see an AGI go from “mildly smarter than human researchers” to “incomprehensible” in just a few years.
I’m unsure which step in that chain of reasoning you think is inaccurate.
You just said “to be honest”, implied I was uninformed because I disagree with you, then said something concurring with me: Zuse’s work happened a decade before ENIAC (1936 vs 1946). If quantum is at the equivalent of Zuse’s work, AI could suddenly accelerate it and go exponential.
https://www.smbc-comics.com/comic/the-talk-3
Go ahead, get scientific.
There is no scientific reason why global quantum communication means that an AI hits exponential growth. (If there is, you sure haven't supplied it! You've just asserted that it is so.)
I understand a number of readers aren’t familiar with QAI research though.
But that’s my point —
> Why should we get scientific? You didn't.
The person responding made a generic negative comment (as did you) due to not knowing the state of the art.
Incorrectly “correcting” people while yourself ignorant is prevented by giving people specific corrections: if you’d taken the time to tell me what you think I’m wrong about, you’d probably realize you don’t actually have a specific thing you think I’m wrong about — you just were surprised by my conclusion.
See here for more detailed arguments:
https://news.ycombinator.com/item?id=26657056
If you were really smarter than me at QM, you’d be able to point out what was wrong with what I said.
The HN downvotes without correction really speak to the hubris around here.
We can conservatively say that even though there is a limit to amount of computation produced in a given volume, temperature range, and time, this limit is still sky-high compared to what we do now. OTOH the speed of light is already a major limiting factor.
There’s a number of facts that make this not sci-fi:
- China has demonstrated distributed quantum networks.
- The US has demonstrated distributed quantum calculations.
- Anyons were recently experimentally verified.
- We have a number of small quantum ALU.
- There are quantum-effect systems in production, just not fully-entangled systems.
- We have quantum ML algorithms for accelerated training.
Nobody has disputed my baseline estimate: that humans will complete a quantum computer in 10-20 years. (If people just disagree there, they should say so.)
However, once you accept that, it’s trivial to see the following sequence:
- AI is slightly smarter than humans, does quantum work 2-5x faster.
- AI develops quantum circuits in 2-4 years that would have taken 10-20 for humans. (See COVID vaccines for power of tech acceleration.)
- Those quantum systems let the AI replace its current systems with faster local ones.
- The AI uses another 1-2 years at this intermediate stage to extend current quantum networking research to unify its quantum data centers into a single computer, giving full global entanglement.
So in 3-5 years, an AI which starts off mildly smarter could finish nearly-done projects and enter an exponential growth curve before we do.
That progression looks like:
- Scale M AI.
- Scale M/K * 2^K, where K is the portion of M at a single data center.
- Scale 2^M AI, when it can fully entangle.
> Because of that, any level of detail is about as believable as any sci—fi.
That’s not true: most major companies in the space have labs looking to turn fundamental research into products, which is the exact opposite of sci-fi.
I think people just haven’t actually thought about this and are responding with fallacies rather than confront an idea they intuitively reject.
Even though I’m right on the technical issues involved.
The central limit theorem shows that the normal distribution emerges as the limit of the average of independent and identically distributed random variables of any distribution.
There are reasons to believe the opposite, that there's a limit. Not just practical technological limitations (e.g. limits to CPU process, speed, power consumption, etc) but also absolute limits for information processing under certain physical constraints (in the Shannon sense).
with strawmen like these, who even needs a debate?
So the way to have a 300 IQ would be to sandbag the rest of the world. Give them all smartphones for instance (or irrigate their crops with brawndo)
Also,
> I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion
I have never heard this assumption in a technical discussion in the modern age, and the author provides no source here. I'm overall unconvinced by this article.
Dogs probably have higher emotional intelligence (by human standards) than wolves, and we probably could breed dogs (or have we?) that are smarter at basic puzzle solving activities like opening latches with their paws or whatever.