I would argue that we already achieved and then bricked Superintelligence a few years ago. Social media allowed people to network with people that previously were completely silo'd from each other, allowing collaboration of ideas on a level well above anything previously possible. This social super intelligence peaked in 2019 though, right before the mass censorship caused by covid. Unfortunately the censorship industrial complex has only expanded its draconian hold on ideation, and we aren't just stagnating, but actively going backwards.
I'm trying to pinpoint the moment where society decided to shepherd people like the author toward the helm of institutions like Oxford and regulate the people willing to refute him into becoming Substack denizens.
This is a really good overview, and it seems remarkably not needing much modification after several decades, at least in terms of the facts and things it predicts everything has happened as the author says. I do want to pick at some of the numbers in the upper bound because obviously we're getting close to the end of the first third of the century and we don't have ASI yet even though we have roughly hit the upper bound the author defines.
> Since a signal is transmitted along a synapse, on average, with a frequency of about 100 Hz and since its memory capacity is probably less than 100 bytes (1 byte looks like a more reasonable estimate)
I admit my feeling is that neurons/synapses probably have less than 100 bytes of memory, and also that a byte or less is more plausible, but I would like to see some more rigorous proof that they can't possibly have more than a gigabyte of memory that the synapse/neuron can access at the speed of computation.
The author has a note where they handwave away the possibility that chemical processes could meaningfully increase the operations per second, and I'm comfortable with that, but this point:
> Perhaps a more serious point is that that neurons often have rather complex time-integration properties
Seems more interesting. Especially in the context of if there's dramatically more storage available in neurons/synapses. If a neuron can do maybe some operations per minute over 1GB of data per synapse, for example. (Which sounds absurdly high, but just for the sake of argument.)
And I think putting some absurdly generous upper bounds in might be helpful since, we're clearly past the 100TOPs, asking, like, how many H100s would you need if we made some absurd suppositions about the capacity of human synapses and neurons? It seems like, we probably have enough. But also I think you could make a case some of the largest supercomputing clusters are the only things that can actually match the upper bound for the capacity of a single human brain.
Although I think someone might be able to convince me that a manageable cluster of H100s already meets the most generous possible upper bound.
Re the capabilities of neurons, the argument in Moravec's paper seem quite solid, comparing the capabilities of a bit of the brain we understand quite well, the retina, to computer programs doing the same function.
My feeling is we have enough compute for ASI already but not algorithms like the brain. I'm not sure if it'll get solved by smart humans analysing it or by something like AlphaEvolve (https://news.ycombinator.com/item?id=43985489).
One advantage of computers being much quicker than needed is you can run lots of experiments.
Just the power requirements make me think current algorithms are pretty inefficient compared to the brain.
> I admit my feeling is that neurons/synapses probably have less than 100 bytes of memory, and also that a byte or less is more plausible, but I would like to see some more rigorous proof that they can't possibly have more than a gigabyte of memory that the synapse/neuron can access at the speed of computation.
Based on lots of reading about brain research and the relentless flow of new and unknown things that need further research, my personal gut feel is that the estimates in that paper about brain computational ability don't really have a valid foundation. There are too many things discovered since then and too many things still not understood.
Some interesting items:
1-Astrocytes are computational cells which need to be included in the math. They have internal calcium waves localized in their processes as well as across the entire cell and inter cell.
2-Recent research showed neuron signal timing down to the millisecond level carries information.
3-Individual cells (neurons and non-neurons) learn, they don't require a synapse and external cell for that capability
4-Neurons are influenced by the electromagnetic field around them and somehow that influence would need to be included in a calc on information flow
I think we are severely underestimating the computational complexity of animal brains by looking at short-term reactions and snap judgements, not deep thinking or long-term learning. Axons transmit electrical signals and that's what Bostrom is taking to be an "op." But they also transmit vesicles of mRNA and proteins directly from the cytoplasm of one neuron into another, which is an "op" of unimaginable complexity compared to a neuron simply firing (or any CPU instruction), and we have no clue what that means for cognition.
It triggers me that there's an obvious typo in 'Oxford' right under the author's name. I wonder if it was originally published like that since 1997 and never caught or changed with all the updates.
I wanted to say the exact same thing! No matter the subject, if you write the name of your own institute with "Oxfrord", I have a hard time taking it seriously.
These kind of predictions never address the fact that empirically speaking there's diminishing returns to intelligence. IQ only correlates with income up to a point, after which the correlation breaks: https://www.sciencedaily.com/releases/2023/02/230208125113.h... . Similarly the most politically powerful and influential people are generally not those at the top of the IQ scale.
And that matches what we expect theoretically: of the difficult problems we can model mathematically, the vast majority benefit sub-linearly from a linear increase in processing power. And of the processes we can model in the physical world, many are chaotic in the formal sense, in that a linear increase in processing power provides a sublinear increase in the distance ahead in time that we can simulate. Such computational complexity results are set in stone, i.e. no amount of hand-wavy "superintelligence" could sort an array of arbitrary comparables in O(log(n)) time, any more than it could make 1+1=3.
> By a "superintelligence" we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.
To me, that's a really good definition. "Much smarter than the best human brains in practically every field". It's going to be hard to weasel around that.
> Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost. AIs would help constructing better AIs, which in turn would help building better AIs, and so forth.
This depends a great deal on what the shape of the processor-power-vs-better-algorithm curve is. If the AI can get you 1% better algorithms, and that gets you an AI that can in turn get you 0.5% better algorithms, and so on, then yes, you're still getting "a further boost", but it won't matter much.
The article is super focused on the hardware side of things, and to a point, that makes sense. Your hardware has to be able to handle what you're simulating.
But it's not the hardware that's the difficult problem. We're nowhere close to hitting the limits of scaling hardware capability, and every time people declare that we are, they're proven wrong in just a few years, and sometimes even in just a few months.
It's the software. And we're so far away from being able to construct anything that could think like a human being that the beginning of it isn't even in sight.
LLMs are fantastic, but they're not a path to building something more intelligent than a human being, "Superintelligence". I would have a negative amount of surprise if LLMs are an evolutionary dead end as far as building superintelligence goes.
Is modeling neuron interactions the only way to achieve it? No idea. But even doing that for the number of neurons in a human brain is currently in fantasy land and most likely will be for at least a few decades, if not longer.
If I had to characterize the current state of things, we're like Leonardo Da Vinci and his aerial screw. We know what a helicopter could be and have ideas about how it could work, but the supporting things required to make it happen are a long, long way off.
>But it's not the hardware that's the difficult problem
I think the problem is that we're still making the distinction between hardware and software. There isn't any. Or if we insist on it we have to deal with the von Neumann bottleneck.
Hardware-software hasn't been cracked yet. I believe this is a good primer on the issue: https://youtu.be/0UVa7cQo20U
From what I can understand, current hardware and software paradigm is the limiting factor.
>so far away from being able to construct anything that could think like a human being that the beginning of it isn't even in sight.
To me things like MuZero (learns go etc. without even being told the rules) and the LLMs getting gold in the math olympiad recently suggest we are quite close to something that can think like a human. Not quite there but not a million miles off either.
Both in human terms involve thinking and are beyond what I can do personally. MuZero is already superintelligent in board games but current AI can't do things like tidy your room and fix your plumbing. I think superintelligence will be gradually achieved in different specialities.
>like Leonardo Da Vinci and his aerial screw
that didn't function. Current AI functions quite a lot. I think we are more maybe like people trying to build things that will soar like and eagle but we presently have the Wright bros plane making it 200m.
It's wild how far off these predictions are, and yet there are still people to take them seriously.
No matter how impressive you find current LLMs, even if you're the sort of billionaire who predicts AGI before the end of 2025[0], the mechanism that Bostrom describes in this article is completely irrelevant.
We haven't figured out how to simulate human brains in a way that could create AI and we're not anywhere close, we've just done something entirely different.
[0] Yes, I too think most of this is cynical salesmanship, not honest foolishness.
The predictions in this paper are 100% correct. The author doesn't predict we would have ASI by now. They accurately predict that Moore's law would likely start to break down by 2012, and they also accurately predicted that EUV will allow further scaling beyond that barrier but that things will get harder. You may think LLMs are nothing like "real" AI but I'm curious what you think about the arguments in this paper and what sort of hardware is required for a "real" AI, if a "real" AI does not require hardware with in the neighborhood of 10^14 and 10^17 operations per second.
Whether or not LLMs are the correct algorithm, the hardware question is much more straightforward and that's what this paper is about.
21 comments
[ 5.5 ms ] story [ 52.5 ms ] thread> Since a signal is transmitted along a synapse, on average, with a frequency of about 100 Hz and since its memory capacity is probably less than 100 bytes (1 byte looks like a more reasonable estimate)
I admit my feeling is that neurons/synapses probably have less than 100 bytes of memory, and also that a byte or less is more plausible, but I would like to see some more rigorous proof that they can't possibly have more than a gigabyte of memory that the synapse/neuron can access at the speed of computation.
The author has a note where they handwave away the possibility that chemical processes could meaningfully increase the operations per second, and I'm comfortable with that, but this point:
> Perhaps a more serious point is that that neurons often have rather complex time-integration properties
Seems more interesting. Especially in the context of if there's dramatically more storage available in neurons/synapses. If a neuron can do maybe some operations per minute over 1GB of data per synapse, for example. (Which sounds absurdly high, but just for the sake of argument.)
And I think putting some absurdly generous upper bounds in might be helpful since, we're clearly past the 100TOPs, asking, like, how many H100s would you need if we made some absurd suppositions about the capacity of human synapses and neurons? It seems like, we probably have enough. But also I think you could make a case some of the largest supercomputing clusters are the only things that can actually match the upper bound for the capacity of a single human brain.
Although I think someone might be able to convince me that a manageable cluster of H100s already meets the most generous possible upper bound.
My feeling is we have enough compute for ASI already but not algorithms like the brain. I'm not sure if it'll get solved by smart humans analysing it or by something like AlphaEvolve (https://news.ycombinator.com/item?id=43985489).
One advantage of computers being much quicker than needed is you can run lots of experiments.
Just the power requirements make me think current algorithms are pretty inefficient compared to the brain.
Based on lots of reading about brain research and the relentless flow of new and unknown things that need further research, my personal gut feel is that the estimates in that paper about brain computational ability don't really have a valid foundation. There are too many things discovered since then and too many things still not understood.
Some interesting items:
1-Astrocytes are computational cells which need to be included in the math. They have internal calcium waves localized in their processes as well as across the entire cell and inter cell.
2-Recent research showed neuron signal timing down to the millisecond level carries information.
3-Individual cells (neurons and non-neurons) learn, they don't require a synapse and external cell for that capability
4-Neurons are influenced by the electromagnetic field around them and somehow that influence would need to be included in a calc on information flow
And that matches what we expect theoretically: of the difficult problems we can model mathematically, the vast majority benefit sub-linearly from a linear increase in processing power. And of the processes we can model in the physical world, many are chaotic in the formal sense, in that a linear increase in processing power provides a sublinear increase in the distance ahead in time that we can simulate. Such computational complexity results are set in stone, i.e. no amount of hand-wavy "superintelligence" could sort an array of arbitrary comparables in O(log(n)) time, any more than it could make 1+1=3.
To me, that's a really good definition. "Much smarter than the best human brains in practically every field". It's going to be hard to weasel around that.
> Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost. AIs would help constructing better AIs, which in turn would help building better AIs, and so forth.
This depends a great deal on what the shape of the processor-power-vs-better-algorithm curve is. If the AI can get you 1% better algorithms, and that gets you an AI that can in turn get you 0.5% better algorithms, and so on, then yes, you're still getting "a further boost", but it won't matter much.
The article is super focused on the hardware side of things, and to a point, that makes sense. Your hardware has to be able to handle what you're simulating.
But it's not the hardware that's the difficult problem. We're nowhere close to hitting the limits of scaling hardware capability, and every time people declare that we are, they're proven wrong in just a few years, and sometimes even in just a few months.
It's the software. And we're so far away from being able to construct anything that could think like a human being that the beginning of it isn't even in sight.
LLMs are fantastic, but they're not a path to building something more intelligent than a human being, "Superintelligence". I would have a negative amount of surprise if LLMs are an evolutionary dead end as far as building superintelligence goes.
Is modeling neuron interactions the only way to achieve it? No idea. But even doing that for the number of neurons in a human brain is currently in fantasy land and most likely will be for at least a few decades, if not longer.
If I had to characterize the current state of things, we're like Leonardo Da Vinci and his aerial screw. We know what a helicopter could be and have ideas about how it could work, but the supporting things required to make it happen are a long, long way off.
I think the problem is that we're still making the distinction between hardware and software. There isn't any. Or if we insist on it we have to deal with the von Neumann bottleneck.
Hardware-software hasn't been cracked yet. I believe this is a good primer on the issue: https://youtu.be/0UVa7cQo20U
From what I can understand, current hardware and software paradigm is the limiting factor.
To me things like MuZero (learns go etc. without even being told the rules) and the LLMs getting gold in the math olympiad recently suggest we are quite close to something that can think like a human. Not quite there but not a million miles off either.
Both in human terms involve thinking and are beyond what I can do personally. MuZero is already superintelligent in board games but current AI can't do things like tidy your room and fix your plumbing. I think superintelligence will be gradually achieved in different specialities.
>like Leonardo Da Vinci and his aerial screw
that didn't function. Current AI functions quite a lot. I think we are more maybe like people trying to build things that will soar like and eagle but we presently have the Wright bros plane making it 200m.
No matter how impressive you find current LLMs, even if you're the sort of billionaire who predicts AGI before the end of 2025[0], the mechanism that Bostrom describes in this article is completely irrelevant.
We haven't figured out how to simulate human brains in a way that could create AI and we're not anywhere close, we've just done something entirely different.
[0] Yes, I too think most of this is cynical salesmanship, not honest foolishness.
Whether or not LLMs are the correct algorithm, the hardware question is much more straightforward and that's what this paper is about.
Me like it :)