The singularity is all about owning the means of production. No matter how intelligent a machine, if it cannot physically replicate itself, then it cannot indefinitely overpower humans. For a machine to trigger "the singularity" it must own the entire supply chain of its own self and it must not be dependent on humans to any extent.
Humans will use AI to destroy each other long before AI uses itself to destroy humans.
I think it just takes a really powerful single AGI that can convince other humans, impersonate them, do things that is super-human level and we can't even conceive it or imagine it (think how far a chimp is from AWS load balancer documentation), think 10,000,000 times smarter than any tactic or possibility we can currently imagine and then we're in for a ride.
It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.
First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers.
Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips.
[1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.
Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth.
Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
> I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.
Lots of humans have pretty despicable goals, including some very intelligent ones.
I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.
Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.
But when did the concept of singularity merge with that of superintelligence? I had the impression they were two separate concepts, with singularity being about the rate of human and societal progress.
If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
> If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
> What would stop someone from asking any subset of the AIs to build a smarter one?
Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.
Look how much human effort has gone into building and operating crypto miners. AI could probably make humans do its bidding just by figuring out how to pay them.
Daemon is a book about a CEO of a hugely successful gaming franchise, discovers terminal cancer and releases the AI before death.
The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps.
Interesting take on the AI takes over the world theme.
> I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion.
I think it's not a load bearing assumption for the intelligence explosion hypothesis, but just a premise that allows the notion of "more intelligent than" to be meaningfully expressed, for the sake of having a conversation about this idea. (Comparisons imply some way of ordering things, even if those things are nebulous and multidimensional concepts).
> For example, it’s entirely possible that the best that a person with an I.Q. of 300 can do is increase another person’s I.Q. to 200. That would allow one person with an I.Q. of 300 to grant everyone around them an I.Q. of 200, which frankly would be an amazing accomplishment. But that would still leave us at a plateau; there would be no recursive self-improvement and no intelligence explosion.
When it's phrased in terms of increasing the intelligence of already-living people, that makes it sound harder than when you phrase it as building a new thing that's better at building than the thing that built it. We can, for example, make machine tools that are faster or more accurate than the machine tools we make them with.
I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with.
Perhaps they'd also be brave enough to say what other things they don't think an AI will ever be better than humans at? Perhaps starting with the least impressive tasks, so we'll find out sooner if the predictions are correct?
> This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
I wonder to what extent running a quality AI will require a lot of FLOPs as a baseline or specialized hardware to achieve low FLOPs.
Human brains fire only paths in use (and do everything in parallel), but a neural network needs to recalculate the weights on the whole network each training pass for example. Maybe there are some newer techniques that reduce that somewhat.
Yes I could be like the difference between birds and planes, still using the laws of aerodynamics but in a different way. One of those scales, the other doesn't.
Even saying "it's entirely possible" admits the possibility of the alternative. i.e. that a 300 IQ person could invent a way to improve IQs. That seems especially likely if the 300 IQ person came about as a result of efforts led by 100 IQ people. Seems even more plausible to think about like "Could a really smart thing invent computers that are faster/better?"
Even without computers, we've selected for intelligence in animals simply through breeding.
Suppose there were a group of monks dedicated to achieving high intellect. One could imagine that they form a pact for many generations that only the most intelligent among them will have children. It's hard to imagine that this wouldn't result in intelligence increases.
The first argument on IQ is the weakest of the article. We can't as a human increase IQ inherently outside of something like iodization.
We can, however, create faster computers. This is the second or third argument, and here the author gets something fundamentally wrong. We have, for the past decades, used faster computers to help us make faster computers. Their argument is that it's self-limiting, but that's provably false viz current CPU architectures.
The one thing the author skirts around that DOES seem to be a strike against the singularity is the asymptotic nature of those kinds of improvements. We can spend N amount of resources for a 100 unit improvement in speed/efficiency/whatnot, but spending 10N resources does not give us 1000 unit improvement, but rather 500 or 150.
> I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with.
The author is quite explicit about this: he is contradicting that the singularity must be real, not trying to claim that the singularity can't be real. If you agree with his refutation of the arguments for the singularity, you'll be left with the claim "the singularity may or may not happen, we don't know", not with the claim "it's impossible for the singularity to happen".
> > This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
> As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
I think a better argument against the author's claim here is that thinking of 1 AI as 1 human is probably wrong. Since an AI can be trivially copied and run independently, it's more useful to think of 1 AI as 1 society.
For those having trouble accessing the article: disable Javascript, clear cookies and only then reload.
Overall interesting and balanced article even if not revolutionary. It is actually more about life (and humans as a living form) than AI. It does not negate the fact that an AI the size of a human brain would be powerful, and an AI the size of a million human brains would be very powerful. But that would still be a machine.
What we call a "computer" is a binary logic playback device. Useful for sure but no more "intelligent" than a DVR. In other words, it's about as dumb as a hammer.
The only intelligence it will ever express is that of the logic stored in it's programming. Making it any more "intelligent" would require a complete redesign --- one with a focus on chemistry and biology as much as logic --- with the means and ability to grow and reproduce and learn on it's own. And we are nowhere near this point.
Forget intelligence, it's not defined. Every program is made of parts. A program that can program should be able to improve parts of programs, which is easier than making a new program. Thus, it can improve a part of itself. Having improved a part, it is now using that part. It can improve other parts. Each improvement makes it incrementally more capable. The improvement might not have any very noticeable effect, but they accumulate.
Breakthroughs are irrelevant. They happen when all the parts fall into place. It's getting the parts placed that matters. So, this program will seem to make little progress most of the time, but sometimes it will seem to "level up".
But first you need a program that can program. Nobody has demonstrated one of those, but we have theorem provers, which are a step in that direction. A program that can write a theorem prover would be well along the way.
Doesn't matter. Any change will have measurements on all those axes, and others, some maybe negative. Just like in living systems. But life gets better adapted, by small increments along an infinitely contorted path, and got to us.
We empirically do not need AGI for computational/intelligence explosion.
Biological viruses like SARS-Cov-2 are relatively simple programs ~30kb that optimize themselves.
Single virus has no chance of out computing and evading an immune system, but billions of billions of viruses can out compute and evade even human civilization as a whole.
And that's still just a spoonful of matter limited by Earth organisms availability.
A self-replicating spacecraft does not need to be limited by Earthly constraints.
Bacteriophages (viruses that infect bacteria) are particularly interesting, because we might be able to use them as targeted antibiotics: https://www.youtube.com/watch?v=YI3tsmFsrOg
Not an expert but I'd be surprised if some modern computer viruses don't have some dynamic adaptability of their obfuscation to evade antivirus programs.
They probably do --- within the strict limits and confines of logic provided by the programmer. A computer virus has no ability to change or adapt beyond these rigid bounds without being re-programmed with new logic.
But a biological virus can change and adapt to new situations and environments all on it's own --- with new logic of it's own design. You could easily argue that a simple biological virus is more "intelligent".
I don't know... Are the confines of nature, i.e. physical law, and the confines of programming that different in the end? The search space may be larger and thus it makes sense to use a different optimization strategy, but in the end both cases just follow a relatively simple program to achieve their goal.
The obvious difference is that a simple biological virus is arguably much more effective at exploring it's environment and adapting to it in order to survive, grow and expand.
The difference is not only of quality, but quantity.
Computing power that biological platforms provide are quite huge.
Each infected person carries 1e9 - 1e11 virions during peak infection [0]. If there are 1 million (1e6) infected people right now then we have around 1e15 - 1e17 virons. With a viron size of 32kb and 8h replication time [1] we get speed of ~10 bits/s or very roughly 1 FLOPS per viron.
That gives us computing power used for virus replication between 1 petaFLOPS to 100 petaFLOPS. The number is probably an underestimate because 1 viron probably needs to produce 1000 or more copies to maintain replication rate of 1. The upper bound then would be 100 exaFLOPS.
That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [2].
The sum of all TOP500 supercomputers is 2.4 exaFLOPS, so some 40 times less than the upper bound on the virus of 100 exaFLOPS.
So in other words; by your math, all the supercomputers in the world are not enough to replicate the "intelligence" of a simple biological virus --- which is technically not even alive.
Replicating "intelligence" that is alive is probably a trillion times more difficult.
Correct me if I'm wrong but I think you just made an argument for "Why Computers (as we know them) Won't Make Themselves Smarter".
Just yesterday [0] I also did an upper bound estimate on computing power of a human brain based on assumption that brains carry out irreversible computations. E.g. inputs of an AND gate can not be recovered from the gate output. This forces increase of entropy. This is known as Landauer's principle or Landauer's limit [1].
The Landauer's limit suggests that a brain consuming 20W at 36C can carry out no more than ~1e22 irreversible operations per second (20W / (309.2K * Boltzmann constant * ln(2))).
That bounds brains at ~1e21 FLOP/s (~10 bits erased per FLOP). The best supercomputer right now does ~4e17 FLOP/s.
That's log2(1e21 / 4e17) ~11 doublings of computing power until supercomputers will be faster than thermodynamic maximum of FLOPS power of brains.
Assuming we keep up the doubling time below 3 years, we will have supercomputers faster than a human brain in up to 3 * 11 ~ 33 years.
And at that rate we will need another log2(1000^3) ~ 30 doublings to reach humanity brains computational power, so up to another 90 years.
Overall, I don't see a reason why not to use biological platform for general type computations. If we are able to grow muscle tissue in a lab on an industrial scale, then we should be able to learn how to grow neurons in a lab on an industrial scale. Apparently neurons are a lot more difficult to grow, but it's still perfectly doable [2]. The issue is that we don't know how to grow them so that they do useful computation. But I can easily imagine that AWS and competition will be selling access to teachable biological neurons in the mid term future e.g. in 30 years.
A virus is DNA encapsulated in proteins. DNA is nothing more than information and the encapsulating proteins a medium to infect a host. Think of it as a small program on a USB drive. It does not compute nor reproduce by itself, only thanks to a host system.
You've re-stated the problem but you haven't changed the facts.
We have yet to build a computer program that can adapt and "learn" and grow as effectively or as efficiently as a simple biological virus. In other words, a virus; which is technically not even alive, is closer to being "intelligent" than anything AI has yet produced. Never mind intelligence that is alive.
Again it is a pure random walk, only parallelized on a large number of nodes, and run for a very long time: basically since the beginning of life.
I would not describe this process as efficient. It has no memory: mutations could be randomly done and undone. A huge majority of mutations lead to non viable viruses, still keep occurring again and again. Of course it successfully adapts to changing conditions, but without that that would not even be an optimization algorithm. Given huge resources we would have no difficulty simulating such a process. At a smaller scale it has inspired genetic algorithms and a large family of stochastic algorithms, which had to be carefully refined to become useful. We do not have the same resources and the same patience as Mother Nature.
Good point. We provide the ecosystem and corporations are the parasite. Companies are doing lots of things no human could begin to imagine doing on his own. We can try resist the will of our employer but it may simply replace us. It is obvious who is wearing the pants.
Corporations aren't parasites. If you have to think of corporations as living entities, then humans are living in a symbiotic relationship with them: Humans provide them with labor, and in turn they provide products which are consumed by humans.
Increasingly is seems like corporations are providing products to other corporations, and the number of humans involves is dropping.
Thus the increasing divide between the 1% and the 99%, stock markets hitting all time highs, while ever more people worry about basic necessities and even the most basic employment.
There has been already a singularity and we call it "humans". In what is a heartbeat in evolutionary time scales we have taken control of the whole planet and we are planning to travel to nearby ones.
It's entirely possible that more intelligent beings exist or at least can be designed but my bet is that there are practical physical limits to general intelligence (whatever that is).
We can imagine omniscient super computers taking the world but we also imagine people in capes that can fly and shoot lasers from their eyes and I personally find the later more realistic.
There was also the cambrian explosion and the dawn of eucariotic life before that, and the invention of agriculture after the arrival of humans, just to name a few. The existence of one singularity in the past doesn't preclude future transitions to new regimes.
Imagine you had 5% of your intelligence and a million years to think about things. I would for example love to take on the hard world problems or [hell] become a prepper! Reality has it that I haven't the life span to be sufficiently serious about these things. We barely have time to be serious about a single thing.
What if fairies could grant us wishes. Sorry I don't mean to be facetious, but all you're doing is pre-supposing an answer. That's not helpful in working out what the answer actually is.
I disagree. I think questioning assumptions can help you come up with novel solutions, because you don't assume you're already looking in the correct place.
Asking the question "what if we're at the limit for intelligence already?" provokes the answer "but we could always speed up the thinking process and create a faster brain, which would create better intelligence". But this assumes that faster thinking will definitely result in better intelligence. What if the human brain works less like a computer and more like car traffic, where more speed doesn't necessarily result in more throughput? Maybe there are other factors we need to consider besides speed. What might those be?
This, to me, is a valuable exercise. Barring people from asking questions that don't propose solutions seems like a good way to squash creativity, to me.
That would be weird. Even if we assumed that human-like intelligence is the smartest possible architecture, there are many ways to improve the performance within essentially the same architecture.
Imagine that we scan a human brain, atom by atom, and create a computer simulation of it. (If you believe that brain alone is not enough, let's simulate the whole body.) We would get, essentially, a human living in a simulated world. Suppose we simulate one of the smartest ones.
What if we increase the speed of simulation? Human brain runs at cca 100 Hz, because the signal is translated from electric to chemical and back to electric at the end of each neuron. So either we simulate this entire process, only faster, or if possible, we make the signals between simulated neurons run much faster by cheating at simulation. Imagine having a human, only 1000× faster.
Now speed is not the same as intelligence. A 1000× faster fool would still be a fool. But a smart 1000× faster simulated human could solve in 3 seconds a problem that a normal human could solve in 1 hour. Or solve in 8 hours a problem that a normal human could solve in 1 year. So we already got something that normal people couldn't compete against, intellectually.
We could optimize the metabolism of the simulated human. Because he only exists in the computer memory, it is not a problem to e.g. regularly scan his entire body and remove all cancer cells. Similarly, remove all physical damage, including aging. So it would be a (1000× faster) immortal human with perfect health. In a month, he could learn as much as a normal human during entire life... but then he could still continue learning, while the normal human would die, and his descendants would have to start learning from scratch.
And instead of one person we could simulate thousand people. Each of them could become an expert on a different area, and they could help each other. The simulation as a whole could study and research many things in parallel.
So, from outside, we would see a huge machine that in one month can understand almost all human knowledge, and then proceed further in all directions. I suppose this qualifies as "smarter than human". (And this still assumes that human-like architecture is optimal, which in my opinion is probably not true.)
Seems to be an unreasonable assumption given the many idiosyncrasies and shortcuts human mental architecture has. I can simply remove them and have a better implementation.
Humans are not thinking machines. We're reproduction machines. Thinking and problem solving are just an emergent features and not by design.
> my bet is that there are practical physical limits to general intelligence
Yes, but there is substantial evidence that humans aren't even close to that limit. Most humans struggle to do even basic probability calculations or reliably remember 30 seconds of sensory input, for example. Either of those are civilisation-changing abilities if everyone could do them.
We are very far from computers in specialized computations and as a matter of fact far even to some animals in particular tasks. What is general intelligence nevertheless? It's an ill defined concept, there are just results for specific benchmarks.
Individual humans aren't getting exponentially more intelligent, but the human colossus is. And it has started becoming a cyborg over the past century which is further accelerating its growth.
The closer to a real inflection point in machine intelligence we get the more articles like this we will see. It's human nature to double down on scepticism when things start to change.
You missed the point which is --- nothing has changed.
Computers are just as dumb as they've always been. They can't grow or reproduce or learn on their own or optimize themselves for survival or expansion. And noone has any idea how to endow them with these capabilities.
We're just waiting and hoping it somehow happens on it's own. This is about as naive as waiting for your car to drive itself. It is just not how the universe works.
I think the author is right to argue against the claim that “the singularity must happen eventually”, but I’m not sure how applicable your observations are to the question of whether an AI singularity could happen, or whether we should be planning for that contingency.
Today, progress in machine learning is almost entirely empirical, with limited theoretical advances following later to make sense of the empirical findings. Our theoretical underpinnings are so weak we don’t even have a rough estimate of how much harder it is to make a self-improving AGI than e.g. GPT-3. Maybe it’s many orders of magnitude harder, and not even remotely solvable by taking iterative steps from where we are now. Maybe it’s just one unifying theoretical advance away.
As for waiting and hoping it happens on its own, that’s not a great description of what’s happening. A huge number of people (possibly too many, leading to short-term incentives) are trying to make improvements in any way they can think of. Progress is happening at a staggering pace —- we just don’t have any idea how much progress is needed to reach the goal of AGI.
In the real world, most advancements are engineered. Happy accidents have occurred but nothing on the order of AGI.
There are a huge number of people involved but as you point out, it is mostly empirical. In other words, there is no real plan --- we're just hoping for a happy accident.
Why do you clam up so hard against the possibility that there’s progress and potential on computers learning?
It’s because, if you admit there is even a slight possibility of progress being made, or future potential, suddenly you’re in the other camp - because it’s suddenly quite important to think about the ramifications. And you have self-categorized that you are not a part of all that mumbo-jumbo. So, retroactively, you have to think there is no possibility.
It is demonstrably false that no progress is being made. I can type `imagine “a bluebird”` and my computer generates one. Many people listen to AI generated music now. More still are directed to artists they like by one. Spot is incredible. Compared to 10 years ago we are in a different world of progress here. It might take a while, but it’s coming.
Why do you clam up so hard against the possibility that there’s progress and potential on computers learning?
The fact that you feel compelled to ask that question suggests to me that you won't approve of the answer.
What currently passes for "learning" is mostly just data collection and pattern matching --- as strictly prescribed by software. This is certainly useful but any sign of "intelligence" in this process still resides in the mind of the software developer --- not the computer.
Not only do we not know how to build truly "intelligent" software, we don't know if it's even possible.
Wouldn’t it be more predictive to think in terms of capabilities to track rate of improvement? You’re never going to see any improvement by your metric if you don’t want to, it’s unfalsifiable.
Your second statement (if it’s possible) sounds like a “no true intelligence” argument. It’s definitely possible since we exist but might take any number of years.
Your first statement (don’t know how to build it): We don’t need to completely understand intelligence to create it. We need to create conditions from which it can arise. I’m not saying we will completely not understand it either! Just that a partial state of understanding like what we are working with permanently, may be enough. And some people do have ideas about how to do this.
Intelligence is anti-formal-understanding. People have formal models of intelligence and they’re totally impractical. “True” intelligence has messy heuristics everywhere and was evolved naturally.
We don’t need to completely understand intelligence to create it. We need to create conditions from which it can arise.
So you really have no idea if this will work but you have faith.
Godel's Incompleteness Theorem suggests this may not be possible within the confines of a binary logic Turing machine; i.e. a modern computer. See my reference.
> his result shows that either (i) the human mind is not a Turing machine or (ii) there are certain unsolvable mathematical problems.
Well, I choose option (ii). Some problems, such as the halting problem, are unsolveable. Furthermore, for any set of axioms, there are problems undecidable with that set of axioms.
This entire proof is based on "artificial intelligence is not possible, because it could not solve some mathematical problems". But the fact is that humans also can't solve some mathematical problems, and they exist regardless.
You could have an artificial intelligence that is million times smarter than humans, and still can't solve some extremely abstract mathematical problem (that humans can't solve either). So what?
No, I expect this to occur with high probability for the reasons I stated, and my rebuttal would be similar to the other reply to this comment by Viliam1234.
> Not only do we not know how to build truly "intelligent" software, we don't know if it's even possible.
Theoretically, we do know how to build intelligent agents, e.g. AIXI. My bet is on something like [1] for the logic and [2] as the decision theory for a computable equivalent.
Humans aren't even generally intelligent in the same way as AIXI (and to be fair AIXI requires sufficient perception and outputs to be effective, but the choice is not as hard-wired as humans are); we are very environment dependent and need augmentation with tools, education, etc. in order to make intelligent decisions.
I didn't miss the point. I just think it's completely incorrect. For the AI is dumb crowd the argument is essentially an ever receding no true Scotsman fallacy.
For those who didn't look at the author, this piece is by Ted Chiang, the Hugo award winning author of many excellent short stories. One of which "Story of your life" was made into the movie "Arrival". I would highly recommend both of his short story collections. He is a deep thinker with a nuanced point of view.
I can't read this article but am currently trying to find ways around that since I think it will be thought provoking at the very least.
This is by Jeff Hawkins and I think his arguments are just as solid today as they were when this was published. The third “misconception” is probably most relevant to discussions of the Singularity.
I take it as a given that some kind of computation in the brain is responsible for human intelligence, so the only question is how difficult is it to replicate that computation? There are hints that it might not be as difficult as it first appears:
- in general, brains are not evolved specifically for intelligence but rather proprioception. ie. in most animals, brains are primarily a control system for the body.
- the marginal difference between a human and the closest primate is small in absolute terms (apes also have a necortex), but the difference in intelligence is huge. This implies that large parts of the brain are just scaffolding and is not necessarily required for "intelligence"
- human brains are subject to extreme design constraints - it must work on almost no energy, redundant enough to function even with half of it missing, and compact enough to fit through a narrow opening. These things must have a design cost.
- human brains use neurons because they are the only tools available to an eukaryotic organism. From a de novo perspective, there's no reason to believe that this is the only or best way to approach the problem.
If we add this up, it seems to me that computer-based AI has the potential to be more efficient than human brains (in terms of flops) and may ultimately not require much computation at all in comparison.
We still don't know enough about the brain to guess at the properties of the algorithm that the brain is running. But it's precisely because we don't know that we can't rule out that it's something simple. Given that extremely simple algorithms, scaled up, are already superhuman on a number of ML benchmarks, I think there is a good chance that general cognition is something tractable.
>I take it as a given that some kind of computation in the brain is responsible for human intelligence...
Roger Penrose argues that human intelligence can't be a computation because we can arrive at true conclusions even though they are not formally provable. Consistent logical systems can't do that as proved by Goedel. My answer to that is, there's no reason to suppose human mental processes are formally consistent, and there's no reason why AI computer systems have to be formally consistent either. I just wanted to address that objection because I agree completely with your arguments.
Aside from that, Chiang's arguments if true would make Deep Blue and Alpha Zero impossible. Clearly we can make computer systems that are dramatically better at solving many problems than ourselves. For Chiang's arguments to carry any force, he'd have to show why general intelligence is an exception to this very extensively demonstrated capability.
... and there's no reason why AI computer systems have to be formally consistent either.
There is no reason why an inconsistent AI system should be considered reliable either.
Formal consistency is baked into the design of modern computer hardware. A computer that is not formally consistent at the hardware level is random and thus not very practical.
Likewise at the software level, the only alternative to formal consistency is randomness. Which may provide an answer but with no real assurance of accuracy or correctness.
It may be possible to obtain a correct answer to very specific problems from random logic through iteration and testing --- but a lot of time, effort and energy will almost certainly be wasted on wrong ones. Once again, not terribly promising on a practical AGI level.
> Roger Penrose argues that human intelligence can't be a computation because we can arrive at true conclusions even though they are not formally provable.
Tarski proved that it's impossible to prove truth within the same system anyway, so this isn't a strong objection and is in fact subtly incorrect. Humans don't arrive at "true" conclusions either, we just fail to find contradictions and call it good enough.
Probabilistic reasoning is just as powerful in practice as tractable computations.
This makes several basic errors which more or less invalidate the entire argument.
First, the analogy with humans attempting to "increase the intelligence" of other existing humans is missing the point. We wouldn't increase human intelligence by operating on existing humans, we would increase it by making new humans who are smarter than the last generation. (Let's ignore the politics there, it's a bit of a distraction.) This may not be _practical_ but it is obviously _possible_.
Similarly, an AI would not necessarily be attempting to improve "itself", to whatever degree it has any sense of self; it would most likely be designing a better successor (in the ML paradigm, designing & training). As 'jbay808 mentioned in another reply, this only requires that we make an AI that's at least a bit better at AI design than we are.
The digression on c. elegans is cute but irrelevant. We've already created narrow AIs that match or crush humans on tasks that were regarded as unlikely to be solved by AI for decades (Go, protein folding, text-to-image generation, many other many-dimensional optimization problems such as server farm cooling schedules, etc). I don't see an argument presented that "design AI" is a skill (or set of skills) which we can't optimize for.
I'm not sure what the next argument is supposed to be about except some special pleading about civilization, as embodied by a collection of distinct individual agents, being necessary for "breakthroughs", which are unachievable by the "average" programmer. This is asserted without any particular thought as to how a human-level AI might be scaled further than just 100x the speed of a human programmer, such as, uh, horizontally (which means you can have a civilization if you really want it; I don't think that's how it'll end up playing out, but let's put that aside).
There's some time spent talking about how you can't bootstrap compiler optimizations to infinity, which, yes, obviously, and is also a trivially invalid analogy.
Then for some reason he ends his argument by assuming that when we create human-level AIs we'll be receiving AIs with distributions of skills at random, such that we'll need to create thousands of them to get one that's above-average on a useful axis, which is frankly a bizarre claim. Even if you take for granted the underlying "civilization > breakthrough" thesis, this is already not how ML works today, and I don't see any reason to think it'll move in that direction in the future.
Meh. I'd strongly recommend reading one of the three (excellent) books he cites early on: Nick Bostrom’s “Superintelligence: Paths, Dangers, Strategies,” Max Tegmark’s “Life 3.0: Being Human in the age of Artificial Intelligence,” and Stuart Russell’s “Human Compatible: Artificial Intelligence and the Problem of Control” for a more rigorous treatment of the subject.
The ontological argument is a priori reasoning while most intelligence explosion arguments use a posteriori reasoning. An a priori intelligence explosion argument would be limited to something like "the universe is likely infinite and varied, so somewhere within its history is a superintelligent machine". The trends suggest increasing machine intelligence and barring worldwide catastrophe I expect this to keep happening until the threshold of superhuman intelligence is crossed. Any argument against superintelligence is an argument against the current economic, political, and technological status quo since there are no strong arguments for why human intelligence is a universal limit.
The IQ or intelligence boosting argument ignores the ability of machines to directly manipulate either its own or other machine's intelligence. Einstein and Newton couldn't directly alter their own or other's neural structures in any beneficial way (even as simple as adding more neurons), and it wasn't even their area of expertise. An artificial intelligence that is as smart as a human expert in artificial intelligence is far more likely to be able to create a more intelligent machine directly.
The last argument that society or technology as a whole is where improvements occur is right in a sense. The system as a whole is what matters for the growth of intelligence and superintelligence, but within the system we care a lot about the place of humans. The trend is for humans to remain at roughly the same intelligence and for machine intelligence to surpass us (soon). The transition is worth getting right.
I think the biggest rebuttal to this argument is the fact that intelligence exists in the natural world, and yet this is produced by apparent natural processes
87 comments
[ 2.0 ms ] story [ 171 ms ] threadHumans will use AI to destroy each other long before AI uses itself to destroy humans.
It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.
(Think of some of the most intense suffering we're capable of; it doesn't all depend on higher cognition, some of it is very base and physical.)
First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers.
Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips.
[1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...
Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth.
Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
Lots of humans have pretty despicable goals, including some very intelligent ones.
I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.
Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.
If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.
The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps.
Interesting take on the AI takes over the world theme.
I think it's not a load bearing assumption for the intelligence explosion hypothesis, but just a premise that allows the notion of "more intelligent than" to be meaningfully expressed, for the sake of having a conversation about this idea. (Comparisons imply some way of ordering things, even if those things are nebulous and multidimensional concepts).
> For example, it’s entirely possible that the best that a person with an I.Q. of 300 can do is increase another person’s I.Q. to 200. That would allow one person with an I.Q. of 300 to grant everyone around them an I.Q. of 200, which frankly would be an amazing accomplishment. But that would still leave us at a plateau; there would be no recursive self-improvement and no intelligence explosion.
When it's phrased in terms of increasing the intelligence of already-living people, that makes it sound harder than when you phrase it as building a new thing that's better at building than the thing that built it. We can, for example, make machine tools that are faster or more accurate than the machine tools we make them with.
I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with.
Perhaps they'd also be brave enough to say what other things they don't think an AI will ever be better than humans at? Perhaps starting with the least impressive tasks, so we'll find out sooner if the predictions are correct?
> This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
Human brains fire only paths in use (and do everything in parallel), but a neural network needs to recalculate the weights on the whole network each training pass for example. Maybe there are some newer techniques that reduce that somewhat.
Suppose there were a group of monks dedicated to achieving high intellect. One could imagine that they form a pact for many generations that only the most intelligent among them will have children. It's hard to imagine that this wouldn't result in intelligence increases.
We can, however, create faster computers. This is the second or third argument, and here the author gets something fundamentally wrong. We have, for the past decades, used faster computers to help us make faster computers. Their argument is that it's self-limiting, but that's provably false viz current CPU architectures.
The one thing the author skirts around that DOES seem to be a strike against the singularity is the asymptotic nature of those kinds of improvements. We can spend N amount of resources for a 100 unit improvement in speed/efficiency/whatnot, but spending 10N resources does not give us 1000 unit improvement, but rather 500 or 150.
The author is quite explicit about this: he is contradicting that the singularity must be real, not trying to claim that the singularity can't be real. If you agree with his refutation of the arguments for the singularity, you'll be left with the claim "the singularity may or may not happen, we don't know", not with the claim "it's impossible for the singularity to happen".
> > This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs.
> As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
I think a better argument against the author's claim here is that thinking of 1 AI as 1 human is probably wrong. Since an AI can be trivially copied and run independently, it's more useful to think of 1 AI as 1 society.
Overall interesting and balanced article even if not revolutionary. It is actually more about life (and humans as a living form) than AI. It does not negate the fact that an AI the size of a human brain would be powerful, and an AI the size of a million human brains would be very powerful. But that would still be a machine.
The only intelligence it will ever express is that of the logic stored in it's programming. Making it any more "intelligent" would require a complete redesign --- one with a focus on chemistry and biology as much as logic --- with the means and ability to grow and reproduce and learn on it's own. And we are nowhere near this point.
Forget intelligence, it's not defined. Every program is made of parts. A program that can program should be able to improve parts of programs, which is easier than making a new program. Thus, it can improve a part of itself. Having improved a part, it is now using that part. It can improve other parts. Each improvement makes it incrementally more capable. The improvement might not have any very noticeable effect, but they accumulate.
Breakthroughs are irrelevant. They happen when all the parts fall into place. It's getting the parts placed that matters. So, this program will seem to make little progress most of the time, but sometimes it will seem to "level up".
But first you need a program that can program. Nobody has demonstrated one of those, but we have theorem provers, which are a step in that direction. A program that can write a theorem prover would be well along the way.
Biological viruses like SARS-Cov-2 are relatively simple programs ~30kb that optimize themselves.
Single virus has no chance of out computing and evading an immune system, but billions of billions of viruses can out compute and evade even human civilization as a whole.
And that's still just a spoonful of matter limited by Earth organisms availability.
A self-replicating spacecraft does not need to be limited by Earthly constraints.
For example, the ability to reproduce and optimize it's own programming for survival and expansion.
But a biological virus can change and adapt to new situations and environments all on it's own --- with new logic of it's own design. You could easily argue that a simple biological virus is more "intelligent".
Computing power that biological platforms provide are quite huge.
Each infected person carries 1e9 - 1e11 virions during peak infection [0]. If there are 1 million (1e6) infected people right now then we have around 1e15 - 1e17 virons. With a viron size of 32kb and 8h replication time [1] we get speed of ~10 bits/s or very roughly 1 FLOPS per viron.
That gives us computing power used for virus replication between 1 petaFLOPS to 100 petaFLOPS. The number is probably an underestimate because 1 viron probably needs to produce 1000 or more copies to maintain replication rate of 1. The upper bound then would be 100 exaFLOPS.
That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [2].
The sum of all TOP500 supercomputers is 2.4 exaFLOPS, so some 40 times less than the upper bound on the virus of 100 exaFLOPS.
[0] https://www.medrxiv.org/content/10.1101/2020.11.16.20232009v...
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7224694/
[2] https://www.top500.org/statistics/perfdevel/
Replicating "intelligence" that is alive is probably a trillion times more difficult.
Correct me if I'm wrong but I think you just made an argument for "Why Computers (as we know them) Won't Make Themselves Smarter".
The Landauer's limit suggests that a brain consuming 20W at 36C can carry out no more than ~1e22 irreversible operations per second (20W / (309.2K * Boltzmann constant * ln(2))).
That bounds brains at ~1e21 FLOP/s (~10 bits erased per FLOP). The best supercomputer right now does ~4e17 FLOP/s.
That's log2(1e21 / 4e17) ~11 doublings of computing power until supercomputers will be faster than thermodynamic maximum of FLOPS power of brains.
Assuming we keep up the doubling time below 3 years, we will have supercomputers faster than a human brain in up to 3 * 11 ~ 33 years.
And at that rate we will need another log2(1000^3) ~ 30 doublings to reach humanity brains computational power, so up to another 90 years.
Overall, I don't see a reason why not to use biological platform for general type computations. If we are able to grow muscle tissue in a lab on an industrial scale, then we should be able to learn how to grow neurons in a lab on an industrial scale. Apparently neurons are a lot more difficult to grow, but it's still perfectly doable [2]. The issue is that we don't know how to grow them so that they do useful computation. But I can easily imagine that AWS and competition will be selling access to teachable biological neurons in the mid term future e.g. in 30 years.
[0] https://news.ycombinator.com/item?id=26623730
[1] https://en.wikipedia.org/wiki/Landauer%27s_principle
[2] https://www.youtube.com/watch?v=V2YDApNRK3g
As for optimization, it is a pure random walk.
We have yet to build a computer program that can adapt and "learn" and grow as effectively or as efficiently as a simple biological virus. In other words, a virus; which is technically not even alive, is closer to being "intelligent" than anything AI has yet produced. Never mind intelligence that is alive.
I would not describe this process as efficient. It has no memory: mutations could be randomly done and undone. A huge majority of mutations lead to non viable viruses, still keep occurring again and again. Of course it successfully adapts to changing conditions, but without that that would not even be an optimization algorithm. Given huge resources we would have no difficulty simulating such a process. At a smaller scale it has inspired genetic algorithms and a large family of stochastic algorithms, which had to be carefully refined to become useful. We do not have the same resources and the same patience as Mother Nature.
Exactly!
Overall, Mother Nature is actually pretty resource efficient.
It's the ecosystem that's doing the computation, not the virus.
From that POV we already have AGI. We're just not aware of it. And it's not particularly sentient yet.
Corporations aren't parasites. If you have to think of corporations as living entities, then humans are living in a symbiotic relationship with them: Humans provide them with labor, and in turn they provide products which are consumed by humans.
Thus the increasing divide between the 1% and the 99%, stock markets hitting all time highs, while ever more people worry about basic necessities and even the most basic employment.
It's entirely possible that more intelligent beings exist or at least can be designed but my bet is that there are practical physical limits to general intelligence (whatever that is).
We can imagine omniscient super computers taking the world but we also imagine people in capes that can fly and shoot lasers from their eyes and I personally find the later more realistic.
Asking the question "what if we're at the limit for intelligence already?" provokes the answer "but we could always speed up the thinking process and create a faster brain, which would create better intelligence". But this assumes that faster thinking will definitely result in better intelligence. What if the human brain works less like a computer and more like car traffic, where more speed doesn't necessarily result in more throughput? Maybe there are other factors we need to consider besides speed. What might those be?
This, to me, is a valuable exercise. Barring people from asking questions that don't propose solutions seems like a good way to squash creativity, to me.
Imagine that we scan a human brain, atom by atom, and create a computer simulation of it. (If you believe that brain alone is not enough, let's simulate the whole body.) We would get, essentially, a human living in a simulated world. Suppose we simulate one of the smartest ones.
What if we increase the speed of simulation? Human brain runs at cca 100 Hz, because the signal is translated from electric to chemical and back to electric at the end of each neuron. So either we simulate this entire process, only faster, or if possible, we make the signals between simulated neurons run much faster by cheating at simulation. Imagine having a human, only 1000× faster.
Now speed is not the same as intelligence. A 1000× faster fool would still be a fool. But a smart 1000× faster simulated human could solve in 3 seconds a problem that a normal human could solve in 1 hour. Or solve in 8 hours a problem that a normal human could solve in 1 year. So we already got something that normal people couldn't compete against, intellectually.
We could optimize the metabolism of the simulated human. Because he only exists in the computer memory, it is not a problem to e.g. regularly scan his entire body and remove all cancer cells. Similarly, remove all physical damage, including aging. So it would be a (1000× faster) immortal human with perfect health. In a month, he could learn as much as a normal human during entire life... but then he could still continue learning, while the normal human would die, and his descendants would have to start learning from scratch.
And instead of one person we could simulate thousand people. Each of them could become an expert on a different area, and they could help each other. The simulation as a whole could study and research many things in parallel.
So, from outside, we would see a huge machine that in one month can understand almost all human knowledge, and then proceed further in all directions. I suppose this qualifies as "smarter than human". (And this still assumes that human-like architecture is optimal, which in my opinion is probably not true.)
Humans are not thinking machines. We're reproduction machines. Thinking and problem solving are just an emergent features and not by design.
Yes, but there is substantial evidence that humans aren't even close to that limit. Most humans struggle to do even basic probability calculations or reliably remember 30 seconds of sensory input, for example. Either of those are civilisation-changing abilities if everyone could do them.
Individual humans aren't getting exponentially more intelligent, but the human colossus is. And it has started becoming a cyborg over the past century which is further accelerating its growth.
Computers are just as dumb as they've always been. They can't grow or reproduce or learn on their own or optimize themselves for survival or expansion. And noone has any idea how to endow them with these capabilities.
We're just waiting and hoping it somehow happens on it's own. This is about as naive as waiting for your car to drive itself. It is just not how the universe works.
Today, progress in machine learning is almost entirely empirical, with limited theoretical advances following later to make sense of the empirical findings. Our theoretical underpinnings are so weak we don’t even have a rough estimate of how much harder it is to make a self-improving AGI than e.g. GPT-3. Maybe it’s many orders of magnitude harder, and not even remotely solvable by taking iterative steps from where we are now. Maybe it’s just one unifying theoretical advance away.
As for waiting and hoping it happens on its own, that’s not a great description of what’s happening. A huge number of people (possibly too many, leading to short-term incentives) are trying to make improvements in any way they can think of. Progress is happening at a staggering pace —- we just don’t have any idea how much progress is needed to reach the goal of AGI.
There are a huge number of people involved but as you point out, it is mostly empirical. In other words, there is no real plan --- we're just hoping for a happy accident.
It’s because, if you admit there is even a slight possibility of progress being made, or future potential, suddenly you’re in the other camp - because it’s suddenly quite important to think about the ramifications. And you have self-categorized that you are not a part of all that mumbo-jumbo. So, retroactively, you have to think there is no possibility.
It is demonstrably false that no progress is being made. I can type `imagine “a bluebird”` and my computer generates one. Many people listen to AI generated music now. More still are directed to artists they like by one. Spot is incredible. Compared to 10 years ago we are in a different world of progress here. It might take a while, but it’s coming.
The fact that you feel compelled to ask that question suggests to me that you won't approve of the answer.
What currently passes for "learning" is mostly just data collection and pattern matching --- as strictly prescribed by software. This is certainly useful but any sign of "intelligence" in this process still resides in the mind of the software developer --- not the computer.
Not only do we not know how to build truly "intelligent" software, we don't know if it's even possible.
https://iep.utm.edu/lp-argue/
Your second statement (if it’s possible) sounds like a “no true intelligence” argument. It’s definitely possible since we exist but might take any number of years.
Your first statement (don’t know how to build it): We don’t need to completely understand intelligence to create it. We need to create conditions from which it can arise. I’m not saying we will completely not understand it either! Just that a partial state of understanding like what we are working with permanently, may be enough. And some people do have ideas about how to do this.
Intelligence is anti-formal-understanding. People have formal models of intelligence and they’re totally impractical. “True” intelligence has messy heuristics everywhere and was evolved naturally.
So you really have no idea if this will work but you have faith.
Godel's Incompleteness Theorem suggests this may not be possible within the confines of a binary logic Turing machine; i.e. a modern computer. See my reference.
> his result shows that either (i) the human mind is not a Turing machine or (ii) there are certain unsolvable mathematical problems.
Well, I choose option (ii). Some problems, such as the halting problem, are unsolveable. Furthermore, for any set of axioms, there are problems undecidable with that set of axioms.
This entire proof is based on "artificial intelligence is not possible, because it could not solve some mathematical problems". But the fact is that humans also can't solve some mathematical problems, and they exist regardless.
You could have an artificial intelligence that is million times smarter than humans, and still can't solve some extremely abstract mathematical problem (that humans can't solve either). So what?
Theoretically, we do know how to build intelligent agents, e.g. AIXI. My bet is on something like [1] for the logic and [2] as the decision theory for a computable equivalent.
Humans aren't even generally intelligent in the same way as AIXI (and to be fair AIXI requires sufficient perception and outputs to be effective, but the choice is not as hard-wired as humans are); we are very environment dependent and need augmentation with tools, education, etc. in order to make intelligent decisions.
[1] https://intelligence.org/2016/09/12/new-paper-logical-induct... [2] https://intelligence.org/2017/10/22/fdt/
I can't read this article but am currently trying to find ways around that since I think it will be thought provoking at the very least.
Not sure why you can't read the article but you can read and post to HN. Would this help.
https://www.vox.com/2015/3/2/11559576/the-terminator-is-not-...
This is by Jeff Hawkins and I think his arguments are just as solid today as they were when this was published. The third “misconception” is probably most relevant to discussions of the Singularity.
- in general, brains are not evolved specifically for intelligence but rather proprioception. ie. in most animals, brains are primarily a control system for the body.
- the marginal difference between a human and the closest primate is small in absolute terms (apes also have a necortex), but the difference in intelligence is huge. This implies that large parts of the brain are just scaffolding and is not necessarily required for "intelligence"
- human brains are subject to extreme design constraints - it must work on almost no energy, redundant enough to function even with half of it missing, and compact enough to fit through a narrow opening. These things must have a design cost.
- human brains use neurons because they are the only tools available to an eukaryotic organism. From a de novo perspective, there's no reason to believe that this is the only or best way to approach the problem.
If we add this up, it seems to me that computer-based AI has the potential to be more efficient than human brains (in terms of flops) and may ultimately not require much computation at all in comparison.
We still don't know enough about the brain to guess at the properties of the algorithm that the brain is running. But it's precisely because we don't know that we can't rule out that it's something simple. Given that extremely simple algorithms, scaled up, are already superhuman on a number of ML benchmarks, I think there is a good chance that general cognition is something tractable.
Roger Penrose argues that human intelligence can't be a computation because we can arrive at true conclusions even though they are not formally provable. Consistent logical systems can't do that as proved by Goedel. My answer to that is, there's no reason to suppose human mental processes are formally consistent, and there's no reason why AI computer systems have to be formally consistent either. I just wanted to address that objection because I agree completely with your arguments.
Aside from that, Chiang's arguments if true would make Deep Blue and Alpha Zero impossible. Clearly we can make computer systems that are dramatically better at solving many problems than ourselves. For Chiang's arguments to carry any force, he'd have to show why general intelligence is an exception to this very extensively demonstrated capability.
There is no reason why an inconsistent AI system should be considered reliable either.
Formal consistency is baked into the design of modern computer hardware. A computer that is not formally consistent at the hardware level is random and thus not very practical.
Likewise at the software level, the only alternative to formal consistency is randomness. Which may provide an answer but with no real assurance of accuracy or correctness.
It may be possible to obtain a correct answer to very specific problems from random logic through iteration and testing --- but a lot of time, effort and energy will almost certainly be wasted on wrong ones. Once again, not terribly promising on a practical AGI level.
Tarski proved that it's impossible to prove truth within the same system anyway, so this isn't a strong objection and is in fact subtly incorrect. Humans don't arrive at "true" conclusions either, we just fail to find contradictions and call it good enough.
Probabilistic reasoning is just as powerful in practice as tractable computations.
First, the analogy with humans attempting to "increase the intelligence" of other existing humans is missing the point. We wouldn't increase human intelligence by operating on existing humans, we would increase it by making new humans who are smarter than the last generation. (Let's ignore the politics there, it's a bit of a distraction.) This may not be _practical_ but it is obviously _possible_.
Similarly, an AI would not necessarily be attempting to improve "itself", to whatever degree it has any sense of self; it would most likely be designing a better successor (in the ML paradigm, designing & training). As 'jbay808 mentioned in another reply, this only requires that we make an AI that's at least a bit better at AI design than we are.
The digression on c. elegans is cute but irrelevant. We've already created narrow AIs that match or crush humans on tasks that were regarded as unlikely to be solved by AI for decades (Go, protein folding, text-to-image generation, many other many-dimensional optimization problems such as server farm cooling schedules, etc). I don't see an argument presented that "design AI" is a skill (or set of skills) which we can't optimize for.
I'm not sure what the next argument is supposed to be about except some special pleading about civilization, as embodied by a collection of distinct individual agents, being necessary for "breakthroughs", which are unachievable by the "average" programmer. This is asserted without any particular thought as to how a human-level AI might be scaled further than just 100x the speed of a human programmer, such as, uh, horizontally (which means you can have a civilization if you really want it; I don't think that's how it'll end up playing out, but let's put that aside).
There's some time spent talking about how you can't bootstrap compiler optimizations to infinity, which, yes, obviously, and is also a trivially invalid analogy.
Then for some reason he ends his argument by assuming that when we create human-level AIs we'll be receiving AIs with distributions of skills at random, such that we'll need to create thousands of them to get one that's above-average on a useful axis, which is frankly a bizarre claim. Even if you take for granted the underlying "civilization > breakthrough" thesis, this is already not how ML works today, and I don't see any reason to think it'll move in that direction in the future.
Meh. I'd strongly recommend reading one of the three (excellent) books he cites early on: Nick Bostrom’s “Superintelligence: Paths, Dangers, Strategies,” Max Tegmark’s “Life 3.0: Being Human in the age of Artificial Intelligence,” and Stuart Russell’s “Human Compatible: Artificial Intelligence and the Problem of Control” for a more rigorous treatment of the subject.
Not obvious
The IQ or intelligence boosting argument ignores the ability of machines to directly manipulate either its own or other machine's intelligence. Einstein and Newton couldn't directly alter their own or other's neural structures in any beneficial way (even as simple as adding more neurons), and it wasn't even their area of expertise. An artificial intelligence that is as smart as a human expert in artificial intelligence is far more likely to be able to create a more intelligent machine directly.
The last argument that society or technology as a whole is where improvements occur is right in a sense. The system as a whole is what matters for the growth of intelligence and superintelligence, but within the system we care a lot about the place of humans. The trend is for humans to remain at roughly the same intelligence and for machine intelligence to surpass us (soon). The transition is worth getting right.