I have a personal belief that I can’t quite articulate in rigorous scientific terms that there is some information-theoretic barrier for us to understand the “true nature” or “essence” of our own intelligence, and so if we can’t get to that, we’ll never be able to model it (notwithstanding “all models are wrong”).
The belief that we can get to AGI comes off as religion to me. It is a substitute for something we can’t really understand, and it will continue to shift the more we learn, yet always remain out of reach. There will be some true believers, and some people simply gunning for power.
The idea that there’s some ‘ information-theoretic barrier for us to understand the “true nature” or “essence” of our own intelligence’ sounds more religious to me.
If evolution can cross that barrier just by banging molecules together and seeing which ones work, it seems unlikely there’s some causal disconnect that makes it impossible for us to get there by thinking about it.
> If evolution can cross that barrier just by banging molecules together and seeing which ones work, it seems unlikely there’s some causal disconnect that makes it impossible for us to get there by thinking about it.
Evolution also crossed the flight barrier by banging molecules together. I don't think banging molecules together without having an understanding of physics and the forces involved would have been a viable means for us to get to flight.
AI/flight analogies are tired, but the OPs argument amounts to the equivalent of, before the Wright Brothers, proclaiming ‘there’s an inherent inability for humans to ever conceive of a way to engineer heavier than air flight’.
It’s a ‘man was never mean to fly, therefore heavier than air flight is impossible’ argument.
But wasn't it through experimentation that we developed that understanding? And in the earliest phases of that experimentation, that's when our understanding was at its weakest, right? You almost seem to be implying that we had the understanding before ever attempting the experiments.
I dont have a strong opinion about it, but when I read parents post I immediately thought of Gödel s works and how there are limits of what you can know within a system (and in Mathematics no less). Thinking that something similar exists in intelligence does not seem so far fetched. As a hypothesis, of course.
It is an analogy to mention that is not a far fetched idea that there are limits.
As far as I recall, one of the results of Gödel is that a system capable of Arithmetic cannot prove certain things about itself. Aka there are limits within a system. My claim was about limits existing.
The reason it seems far fetched is we have one example of a mechanistic, physics based machine that we know can perform the computations necessary to produce intelligence.
I could understand the hypothesis that AGI is not computable, if we didn’t have an existing example of a machine that can produce AGI.
Since we do have an existing machine that can produce AGI, we would have to suppose it does something:
- outside of physics to achieve its results
- performs some operation that is impossible for us to understand or replicate
The article of faith is not that the universe is a computer. The "article of faith" is that the universe follows physical rules.
I also don't think it's fair to call it an article of faith, since we have strong evidence to show that _to date_ the universe has followed predictable physical rules. That could, obviously, change at any time, but it seems at least a reasonable prior to assume that the physical rules that we've studied in the past will continue to operate into the future. "The sun will rise tomorrow" is, I _guess_ an article of faith, but I think it's more fair to say it's a reasonable and well founded prediction based on a well studied model of the solar system and the physical laws we've observed the universe follow in the past.
So, my beliefs are:
- The universe is mechanistic, and follows physical rules
- Human beings are also mechanistic, and follow the same physical rules as the universe
- Human beings are intelligent
- Human beings are at minimum turing complete computers (since you could give me a roll of paper, a set of op-codes, and I could perform calcuations)
So, it seems to me a reasonable starting assumption that intelligence is a result of the mechanistic universe. Do we have any evidence for something outside the physical, mechanistic universe impacting human cognition? I'm not aware of any.
But we have lots of evidence of the physical, mechanistic universe impacting human cognition and intelligence.
I'm willing to acknowledge that we don't know that intelligence is Turing Computable, but I'd argue that intelligence is likely to be a mechanical process that's compatible with the physical rules of the universe. Can we be certain of that? No, we can not be 100% certain. But it seems a much more reasonable and evidenced hypothesis than something which asserts there is a metaphysical process that produces intelligence, but which we have no evidence for.
So, no, I don't think it's reasonable to say that assuming a "mechanistic universe" is an article of faith. I think it's a reasonable belief, based on the evidence that we have. What would be an article of faith is asserting that it could _only_ be a mechanistic universe, and refusing to accept any evidence to the contrary. I have not done that.
I think a mechanistic universe is not necessarily a computation, so purely physical intelligence is also not necessarily a computation. But I'm not prepared at the moment to go dig up the alternatives to "computation" and "metaphysical process", or to figure out how a deterministic (or even stochastic) process could be uncomputable.
> so purely physical intelligence is also not necessarily a computation.
We're going to get into semantics pretty quickly, but I would argue that at least purely physical intelligence is a procedure that is capable of being run on a machine that exists in our universe.
I would also argue that, such a machine is _at least_ turing complete (since I would think a general intelligence should be capable of emulating a turing machine in the same way that humans are able to do so). I would happily accept that it's possible that being turing complete is a necessary, *but not sufficient* condition for intelligence. That is, I could see a world where intelligence requires some form of hyper-turing machine, that is able to solve some non-turing computable problems.
However, I would argue that even if intelligence does require non-turing computable functions, there exists *a* machine which can perform those procedures (that is, we exist). Thus, in this hypothetical universe where intelligence is non-turing computable, we would then have an existence proof for a hyper-turing machine which can compute non-turing computable results. In this universe, then, I'd argue that anything that can be produced by this "hyper-turing machine", _is_ "physically computable", even if not "turing-computable".
Ultimately, if we accept the premise of a mechanistic universe, I think humans are an existence proof of a machine capable of producing intelligence. Then, I think it follows that it *cannot* be physically impossible to create a machine capable of producing intelligence. Whether we call that machine a "computer", though, I don't have a strong preference.
Neither humans nor any physical device is Turing complete; only bounded computations can we perform. What do you mean by mechanistic? Mathematical? Deterministic? Non-dual with no unmeasurable causes?
So, I’ve only used it in a “I understand what it means and what I mean by it” manner in the past, and I haven’t rigorously defined it for myself.
I’ll do the best giving an off-the-cuff definition, but I’m sure it will have some holes and would be improved with some time spent thinking about it, or reading to crib a definition from someone else who has already put in that time.
I think my definition of a mechanistic universe would be:
A universe that transitions from one state to an adjacent state through a consistent set of rules.
From your options:
- mathematical? I’m think yes, but potentially only if “mathetics” is defined broadly.
- Deterministic? I don’t think this is a requirement for a mechanistic universe. My bet would be on a deterministic universe (through an Everettian interpretation lens), but I think something can be mechanistic while being stochastic, so long as the rules determine the probabilities. That is, I would consider a computer with a true source of randomness to still be mechanistic.
> Non-dual with no unmeasurable causes?
Non-dual, yes. That is the primary meaning for me. “With no unmeasurable causes” is a phrasing that makes me slightly uneasy. I could imagine a situation where, if the universe is stochastic you could end up in a situation where you cannot precisely identify the specific cause for a specific event. But, I think as the spirit of this question is intended, yes.
Thanks, that all makes sense. Do you follow Sean Carroll's takes on many worlds? A very consistent philosophy of reality, and some interesting math starting to come out with space-time itself being an emergent feature of Hilbert space evolution of the wave equation (where it is constructed essentially out of varying degrees of coupling between different states). Even a paper where this sort of coupling-derived space might obey the Einstein equations.
I listen to his podcast, and I’m definitely partial to the many worlds interpretation, so I end up taking in a lot of his explanations of it.
The work he’s doing in that area is super interesting, and sounds promising (of course, I’m getting primarily Carroll’s take on it when I hear about it, so it makes sense that it sounds promising!), but the math itself is above my head.
I am super interested to see if it ends up moving the ball forward on resolving the GR/QM conflicts.
I personally don't think it will but I guess it could, it's as good a model to start from as any. That's kind of the problem though, it's not actually better than the modern version of the Copenhagen interpretation unless you think wave function collapse is a physical event that's caused by (something in) your brain. If you think that, Copenhagen's obviously not correct, as your brain can't cause anything that it itself wasn't caused to do, because free will does not exist. If you think of wave function collapse as not a physical event and not caused by anything (except by the physics of the universe following the original state of the universe, if you don't like the idea of a block universe in configuration space), it's totally fine and is equivalent to Multiple Worlds. Don't think of the Multiple Worlds as things that physically exist either though.
Can we stop with this stuff? It is a 100% useless distinction, Turing machines can only ever use a fixed amount of memory, so we can run any Turing machine for as long as we want to, we can just pause it, “change the tape” to a bigger one, and continue.
Well, it's not useless mathematically. The Turing machine cannot run on all inputs with a bounded tape size. The quickly growing functions show pretty strongly that no physically incarnateable device can compute easy to specify functions on inputs less than 1000. "Turing machine" or Turing equivalence are pretty precisely defined mathematical concepts, so using them in different senses is a bit perverse; in my mind, more perverse than using literally to mean figuratively. Also, how does one change the tape in ones brain? I'd love a pensieve but those also are not physical, at least not yet.
The thing I'd really like is a Turing machine where the nth transition took 1/2^n seconds, so we could run them to infinity in just two seconds.
> So, it seems to me a reasonable starting assumption that intelligence is a result of the mechanistic universe.
> Do we have any evidence for something outside the physical, mechanistic universe impacting human cognition?
Problem is, even granted the above, that's not enough of an argument that AGI is only a matter of mechanism.
Take the analogy with pain. We've discovered the mechanism for pain. That it's e.g. some stimulus applied to skin which sets of nerve signals that register in the brain. To then say that the experience of pain is the same as the physical mechanism we've discovered still misses a key step:
what does the experience of pain inhere to, and is that the same subject as that for the mechanism of pain?
Comparing AGI and human intelligence has the same problem. We don't know exactly what is intelligent in either case, let alone whether the two are comparable. So it's not a question of intelligence per se, but of that which is intelligent. Maybe AGI, the way we are thinking and talking about it, is unavoidably tangled with having to grapple with [self-]consciousness.
Do you think the self-consciousness that we possess is the result of physical processes going on in our brain and body, or do you think something outside of physics is required?
If I made a careful atom by atom (or quark/lepton/boson) copy of every atom in your brain and body in a different location, would that new copy be intelligent?
If our intelligence is purely a result of physical processes, then why do you suppose it would be impossible for us (or more advanced future beings), to construct a machine which can also follow those processes?
If you think your intelligence is not reducible to purely physical processes, then… what is this meta-physical thing that is required for intelligence? Does it interact with anything other than intelligence, or is it only involved with intelligence? Is there any reason or evidence we should assume this meta-physical intelligence?
> ..., or do you think something outside of physics is required?
I am most sympathetic to the argument that self-consciousness is an emergent phenomenon that is not easily modeled, but is nevertheless derived from physics. In the context of my previous comment, it is this emergent phenomenon that is deemed "intelligent".
> If I made a careful atom by atom (or quark/lepton/boson) copy of every atom in your brain and body in a different location, would that new copy be intelligent?
Yes.
> If our intelligence is purely a result of physical processes, then why do you suppose it would be impossible for us (or more advanced future beings), to construct a machine which can also follow those processes?
I think it is possible, and not too far off to boot. We might get (maybe we already have gotten) the intelligence without the particular emergent phenomenon of self-consciousness that is of the kind humans possess, at least not at first. Some of us will call that AGI, some of us won't. Because we are still working out the terminology and taxonomy. Discussions like these in HN help.
Maybe we'll get the emergent phenomenon soon thereafter. If it is in fact intractable to model (and therefore to train for directly), it may happen unpredictably. Exciting times.
Right. I think a charitable reading of the post is something like, "it probably takes a mind superior to a human's in order to willfully develop a human-rivaling AGI".
There's a lot of ground to cover between AI as increasingly elaborate magic 8-ball toys and a real human-rivaling AGI. That is, one capable of observing an environment, identifying goals and problems, planning, acting, and reacting. In these much longer (stateful) cognitive chains, there are more opportunities for pathological failure modes and less opportunity for a toy user to charitably excuse the misbehavior.
This is not some kind of dualist metaphysical argument about the possibility of AGI in the abstract. Merely a doubt that we can blindly scale up the complexity of a synthetic mind to meet or surpass our own. Consider that we still can't even begin to understand our own minds in enough detail to reliably predict, repair, or augment them.
I am outside this field and so may have too much of a layman's perspective. But it seems to me that contemporary AGI believers conflate training and evolution. That nature did it in eons doesn't argue that we can do it in practical product development cycles, unless we can simulate these evolutionary processes to follow a similar search in a compressed time scale.
As a crude analogy, I think todays LLM products are a bit like horoscope generators. Clever arrangements of words that attract a charitable or gullible reader. But AGI use cases are more like wanting a life partner who will understand and willingly assist ones efforts.
That's wrong, see See Gödel, Escher, Bach [1979, pp. 452–456] If you get rid of the infinities and approximate (including cutting recursions short), you can (approximately) compute a whole lot. Not everything, but everything your brain can, and more.
It's wrong that an AI system is more limited than a human brain. Obviously there's computability and Busy Beaver stuff, as well as other earlier practical limits to computation, but that's not a limit to "general intelligence" if you mean the level of a human brain. Human brains approximates and hardly want to recurse at all, and there's no reason an AI can't do the same thing.
If I understand correctly you state that it is wrong to say a computer is more limited than a human brain.
IMO that is far from clear. In some aspects (multiplying matrixes) it might be more powerful. In others (power consumption) the human brain wins by a wide gap.
No GP's point is that Gödel's theorem has nothinhg to do with this debate because the things that Gödel says are impossible aren't the things GPT-9 would need to do to reach a human-level intelligence.
The point that brains are more efficient than computers is valid, but that doesn't tell us much about eg whether we'll be able to make a computer do everything a human brain can do within 50 years.
Sure, and we have pretty good evidence of the mechanistic operation of the universe from quantum mechanics up, and we haven’t found any evidence of anything outside of that.
So, to date, a mechanistic view of the universe seems like one that is the most supported be evidence at the moment.
Have we ruled out meta-physics or some kind of duality? No, of course we haven’t. As long as there’s unknown physics, it’s a possibility.
But I think if you’re reasoning based on the evidence we’ve been able to collect to date, then you should absolutely favor a mechanistic universe to a dual universe.
Intractable and impossible are very different claims.
NP Hardness is a statement about the asymptotic difficulty of solving a problem at ever larger scales. It says ‘if you have a way to solve this problem at size n, that way will scale worse than polynomially when you try to apply it to a problem at size 2n’.
Which might place limits on the practicable scale of how big an n your approach gets to work for. But if your approach works practically for a big enough n to make AGI then your approach works - NP Hardness doesn’t matter.
And since we know that finite mass lumps of finite numbers of gray cells are capable of GI, we have a reasonable expectation that there is some n for which AGI might be possible.
Yes, If you look at the power efficiency of the brain relative to LLMs, you can see nature has done something special, and I don’t think we should be so quick to call it game over. It might be that LLMs are just some atom or building block of a much larger and more complex thing.
Yes, this is huge. Of all the places to shim in my ¢2, as the topic could go on forever.
I remember talking to a robotics specialist and his take on efficiency was that despite the obvious speed and strength that machines have over us, you need to factor in just how little power is consumed when we move our extremities compared to what is needed for actuators and motors of even the best bots.
Now this was years ago and things could have taken a huge turn, but I think given the recent statements by Altman about the looming power crush drives it home, that even if we do manage something truly monumental, it'll take a fusion reaction to pull off.
I mean, I wouldn’t bet on that either. We built the “computing” empire on silicon, but we might be able to very significantly improve on this with a single discovery/invention. All the computational stuff we built/figured out will move out without trouble.
As a counterpoint, I feel like the belief that we can't get to AGI comes off as religion. It presupposes an ineffable quality that we posses that machines cannot. The argument that we have to fully understand something to build it doesn't hold water for me. I have made plenty of things where understanding the depths of what I had made took far more time and effort than building the thing in the first place.
It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.
Does gold have an 'ineffable' quality that hydrogen does not? Do cells, after 1bn years of evolution have a quality that gold does not? Does reality work one way, and not another? Are not all things unequal?
A "machine" is what, exactly? Do we take it to be an abstraction? Or is it an electrical field oscillating over silicon? Either way, you're in trouble. Abstractions have no physical properties, and electrified sand seems hardly to possess any interesting properties.
The ability for animals to adapt to their environments, by growing into them, by establishing plastic causal connections in their very bodies, grown by their environemtns... able to almost instantly move from protein expression in 1trn 1-bn-yr cellular supercomputers in each of our bodies to macro sensory-motor representation --- and back again
Is this ineffable?
Or is this extremely effable. Is rather, not the superstitious view that "everything is anything" ?
All extant, knowing, studied intelligent systems have organic properties; and radically so. Insofar as this is "ineffable" you should take that up with the animal kingdom.
I find the contrary supersitious, magical, religious, ineffable... that mere abstract patterns in arbiatrily chosen aspects of our bodies are necessary and sufficient conditions for anything at all. This would be the only case in all science. The only physical property instantiated by mere arrangement at any level. Upload our consciousness? Make it out of wood why not!
Nonesense. When I am hungry, I dream of food, when I dream of food I plan to get some, and I am angry without it. This will not be made out of sand.
Given that you need some pretty complex elements to be able to make life in the first place (not to mention complex molecules made out of those elements) I'd say yes, there is definitely an 'ineffable' quality that life has that Hydrogen does not (to short circuit your chain of reasoning), but intelligence isn't necessarily made of a particular kind of matter and that's where the analogy breaks down.
Downvoted for saying reality has physical properties? Where else but hackernews? Sorry to disappoint, but all the algorithms of CS, which do anything at all, have devices in them. ie., their critical steps are impure, and cannot be specified mathematically.
There is nothing abstract in the world, everything is concrete. Discrete mathematics is not some divine realm which is where the Mind really occurs, and the Soul is really kept.
No graph traversal algorithm cares, nor will ever. Science is the study of reality, not computer "science". There are no computational "properties". All interesting properties are of objects extended in space, and in time.
The world is physical, as described by physics; not abstract, as describe by mathematics. And so, not computable.
The proposition of artificial intelligence, whether at human-level or not, is that a suitably-programmed computing device (as opposed to an immaterial algorithm) could manifest the properties which we have chosen to call "intelligence" when we observe them in biological agents, and similarly for consciousness.
Particularly with your comment "no graph traversal algorithm cares, nor will ever", you appear to be trying to impute that the concept of artificial intelligence is predicated on a category error. If so, then this is a fairly common argument, but is itself predicated on a category error.
You don’t have to believe AGI is impossible to believe we have no route to get there. The burden of proof on “we have a way to do this” falls heavily on the person proposing it. And being able to do something that wasn’t expected doesn’t imply the ability to do a specific other thing. It’s that belief that’s I’d say is perhaps more cultish than religious.
This logic doesn’t work with anything else. If I say we’re going to fly to Jupiter you’re going to ask me for specifics not just presume that I know what I’m talking about. Or any of the other infinite things I can claim to be able to do. Why is this one different?
Just to close the metaphorical loop here, the current AI debate is something like:
"There's a danger we'll be able to fly to Jupiter and bring home a Jupiterian life-form that will destroy humanity. Therefore we should stop all space exploration."
The doomers take the equation with some wildly improbable step that we can't currently explain or justify, multiply the outcome of that step by "infinitely bad", and conclude that the whole thing must be dangerous. But to amp up the rhetoric, they always skip over the "wildly improbable" part.
I guess the disagreement you have with doomers is the probablility you assign to us being able to create human level AI. For the record most "doomers" don't pull a pascal's wager here. Standard EA doctrine is that if we stagnate technologically at our current level for too long (read centuries) then we will inevitably be wiped out. So they want technological advance, they just want caution.
I don't know what "most" do, but I see that form of argument all the time -- particularly from the highest profile doomers.
Personally, I think this line of argument is driven by the hype cycle more than reality. Use chatGPT or midjourney or whatever for a while, and it's pretty easy to see that we're dramatically overweighting theories of AGI risk, and dramatically underweighting stuff like "disemploying the bottom 80% of the intelligence bell curve" with technologies that automate away lots of formerly white-collar labor.
If I had to put my money on an "existential risk" attributable to AI, it would be economic strife.
I’m saying that there are very many things that I can’t prove are impossible that I don’t worry about happening. I’ve learned to live with that uncertainty.
Hmm. When the first basic black powder rockets were invented, would it have been unreasonable for someone to think that they could one day reach the moon? Of course there would be many uncertainties -- including what the moon even was. No one could say what technology would be required. But clearly there was a major paradigm shift that suddenly made it within the realm of consideration.
I imagine it might have been unreasonable to suppose that, yes. The fact that the future is unpredictable doesn’t mean that all claims about the future are a priori equally valid.
In the future where AGI takes over the world next year I might look silly for arguing with you about it, but that’s a risk I accept based on the fact that I don’t think that’s going to happen.
We absolutely have a way to get there. A sufficiently large neural network can approximate any function (the universal approximation theorem), which has been mathematically proven. LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it. Eventually we'll have LLMs trained on visual and audio input too (and be trained to predict what happens next), so they can receive an arbitrarily large amount of training samples. If the models keep growing and being trained on more and more data, we'd expect them to become at least as intelligent (good at modelling the world) as humans.
> LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it.
I don't see any evidence that this is true or proven in any way.
Also, the ability to brute force something doesn't mean that brute forcing it is easy or even feasible. You can apply the same logic to "calculating a private key from a public key" that you are to human intelligence. Sure, a large enough neural network can do either, but that doesn't mean that building them is actually realistic.
It has been proven in much simpler toy models that LLMs just trained on token prediction can and do build "world" models[0]. That's not the same as evidence that they must build a world model, but it's proof that at least sometimes they do so even when just being trained on prediction.
This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function. So, there is no absolute certainty that they must be able to approximate human cognition, since there is no reason to believe cognition is continuous.
Also, there is no proof and no reason to believe that any current architecture and, more importantly, that the currently known training algorithms can achieve anything close to cognition. After all, animals and humans seem to learn much much much faster (much smaller training sets) than any algorithm we have so far.
This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function
Incorrect. That's what nonlinear activation functions are for.
Researchers wasted many potentially-fruitful years because Minsky and other luminaries -- people who occupied the same position of authority that LeCun occupies today -- made a huge deal about how the original perceptron and subsequent multilayer variants "couldn't learn XOR." That was almost literally how they put it. At the first sign of difficulty they abandoned the approach that was, and remains, the most promising. We have to be careful to learn from that mistake.
You have to be a lot clearer about what you mean by continuous here. An LLM technically does not produce a continuous function on the reals because its inputs are floating point numbers with finite precision. Any function on discrete inputs like this has an extension to the reals which is continuous, just imagine joining all the discrete points up with lines.
So then your claim wouldn't be about the limits of LLMs themselves, but on the limits of systems that do not take continuous inputs. The question then is do you think that humans take in continuous input?? Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.
It is quantum which isn’t exactly discrete. You receive one photon or two, not 1.5, sure, but the energy of that photon, its frequency is a real. At least I am not aware of a quantum mechanics over Q. I think the math would be hard because you loose all the convergence properties. Maybe there is some Hilbert space over computable reals, Google is not finding it but I no longer find that to be indicative of anything.
I mean continuous in the calculus sense, as in it doesn't have discontinuities, not as in continuous VS discrete.
> Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.
This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.
But continuous depends on the topology of the space you are working over. If the topology is discrete any function is continuous. I've found some papers that argue that simply from an error corrections standpoint the signals in the brain need to be discrete.
Given a fixed level of precision for input and output, and a function on this discrete space, we can construct a continuous extension of this function to the reals. Now using the paper we know that there is a neural network with continuous weights that approximates this continuous extension to arbitrary precision. If we imagine rounding the continuous input and output to the fixed precision specified, then because of continuity of the neural network, and the fact there is a finite number of weights, we can choose a tolerance by which we can change each of the weights such that the output does not change by more than the precision of the output value. Thus we can pick a level of precision for the weights where both the weights and inputs and outputs are discrete.
There is a leap from "predicting the next word" to "modelling the world" that you are sort of glossing over here.
The idea that LLM's are actually "reasoning" rather than just performing a probability function seems a little bit...pseudo-religious. Like we've created a new life form.
Predictive models need not at all reflect what actually happens. Sometimes embracing less accurate models is actually need to enhance understanding. Epicycles in the geocentric model could be made very accurate as far as observations in the sky are concerned, but it wasn't a good world model.
That whitepaper operates with mathematical, infinite precision numbers.
It’s just a categorical error to read too much into it - it’s not really a too interesting property to be able to get closer to something forever, without some sane growth function.
When people say we can't get to AGI, they're typically either in the camp that there's something intrinsically special about us that is incapable of being replicated, or various versions of the belief that it's too complex to be achieved within a timespan in which they would still recognize humanity.
The former can be seen as a religious belief, but the latter is simply saying that we do not understand anywhere near enough about intelligence to develop AGI.
You point to the unexpected leap in capability we've had recently, but fundamentally it isn't that different from what we have had for decades. The same fundamental unsolved limitations exist, such as of being unable to learn the way we do (eg if a child is writing a specific letter wrong, we correct that specific mistake, but that doesn't overwrite the knowledge of how to write the other letters).
As a lecture I watched a while ago had put it, we've gotten pretty good at making the icing of the AGI cake recently, but we still have no idea what the cake is supposed to even look/taste like, so we barely know what ingredients we might need to make it let alone the actual steps involved.
>not understand anywhere near enough about intelligence to develop AGI.
LLM training is essentially training them to predict what happens next. When we combine this with multi-modal training (video/audio), they're being trained to predict what happens next in the world. As we feed them more and more data and make them bigger and bigger, they'll get better and better at this task. Given that predicting what happens next requires predicting what humans do, because humans are part of the world, if they keep getting better and better then they'll be able to predict what humans would do well enough that they're capable of thinking any thoughts humans could think, because that's a requirement for predicting human behaviour. So just by training them on this one label, predict what happens next, we'd expect them to eventually develop human-level intelligence if the loss keeps decreasing.
It may result in something that is even better at faking the appearance of intelligence, but LLMs and similar stuff fundamentally lacks various features that make them not capable of becoming human level intelligences.
Humans are pretty good at faking intelligence too! It's only when you question them carefully and you yourself are an expert in the field that you can detect that they're confabulating.
Conversely, I've never seen anybody produce anything but the next word coming out of their mouth. Therefore people are glorified stochastic parrots.
What do you mean by learning and metacognition? An LLM can tell you whether or not something is in their context window. Humans regularly come up with "innovative" things that they actually saw from somewhere else and just happen to forget the source.
Intelligence is like magic - every time a machine can do one thing on the list, it stops being intelligence, it's just engineering. Maybe intelligence really is just a big bag of tricks?
We don't actually know that yet. It may very well fundamentally have all that is necessary for intelligence but other pieces are missing.
Frontier neuroscience and functional processes for how our brains work has more in common with the fundamentals of LLMs than not, vis-a-vis minimization of prediction errors.
I don’t think “fundamentally having something” is actually meaningful here - as it’s enough to show it is Turing complete for it to have fundamentally everything. Nonetheless, it’s probably not excel that will become the next AGI.
If the loss keeps decreasing forever, then that conclusion seems to follow. But it's likely there will be plateaus and ceilings that current neural net architectures cannot surpass. We likely need more breakthroughs to achieve enough generalized reasoning, reliable predictions, alignment, and scalability within hardware constraints.
> Given that predicting what happens next requires predicting what humans do, because humans are part of the world, if they keep getting better and better then they'll be able to predict what humans would do well enough that they're capable of thinking any thoughts humans could think, because that's a requirement for predicting human behaviour.
This doesn't follow. A just as likely (or even more likely) prediction is that they will fail to predict what happens next once it requires modeling humans. It won't get any more progress even if we increase training set size or model size - it will require an architecture change of some kind.
Yes, it's completely possible that we get to some sort of limit and the scaling laws fall down, loss stops dropping even with more data, parameters, compute. But we have zero reason to believe that is the case, currently, it's only hypothetical. In terms of what we've experimentally seen so far, it just keeps getting better as you scale them up, so to find out if you're right and there's some future roadblock we don't know about we just have to keep scaling these things to see if it works. We haven't even seen loss functions start to level out - the closest argument I've seen to that is people misinterpreting the SGD curve (an artificial tapering we apply to the training process) as a fundamental architecture limitation rather than just an attempt to use resources wisely on a given training run.
Basically, training of these models currently only costs a few million dollars, peanuts compared to the budgets of CERN or ITER, and we have zero evidence that we're approaching any sort of ceiling. If we keep scaling, maybe we see some heretofore unpredicted failure that breaks scaling laws like you're predicting. But maybe we don't, and the scaling laws (which have performed incredibly accurately so far) really do hold.
>Yes, it's completely possible that we get to some sort of limit and the scaling laws fall down, loss stops dropping even with more data, parameters, compute. But we have zero reason to believe that is the case, currently, it's only hypothetical [Edit:- and not really a good hypothetical, because there's no theoretical backing that explains current progress but predicts a future discontinuity]. In terms of what we've experimentally seen so far, it just keeps getting better as you scale them up, so to find out if you're right and there's some future roadblock we don't know about we just have to keep scaling these things to see if it works.
Also I have no idea why you think Occam's Razor supports "There will be a discontinuity in this currently accurate observed trend, near enough in the future to be relevant" when the trend is talking about the effects of an algorithm applied to data. By analogy, it's something like claiming Moore's Law won't hold because Occam's Razor suggests at some point there's a physical breakdown, except instead of saying it now you're saying it in 1978, & you're saying it about an algorithmic output rather than a physical thing with an obvious limit (atomic size). Once we see any evidence of all of such a discontinuity, we can adjust what we're doing based on that. Until then, if we want to know whether scaling laws hold we should scale the models.
First of all, there is no theory at all which explains the current progress of LLMs. It's not even a quantifiable progress - it's clearly there, but no one can say with any true precision whether GPT2 to GPT3 to GPT4 is following linear or exponential or whatever else precise function of progress in capabilities. We can clearly see that they are improving, but we have no idea how to accurately quantify this space.
Secondly, we can see that there are obvious quantifiable physical increases in the requirements of training and inference for GPT4 compared to GPT3 and to GPT2. So, we can absolutely make a physical argument: at some point, you'll need too much energy to keep scaling up GPTs for it to be feasible, even if capability would keep improving with model size.
Thirdly, the GPTs require ever more data to train, and there is a limited amount of text data in the world, of which they are already using a significant percent. So, there is also an argument that we might run out of data to improve even before we run out of physical resources.
And making an argument that Moore's law will stop at some point due to physical constraints was just as correct in 1978 as it is in 2024. Predicting when that is is obviously hard - and the same is true for LLM scaling. We may be at the peek, or there may be 100 more generations before we even approach the limits I'm contemplating. But the limits are certainly there.
This conversation started with someone saying that despite LLMs having a theoretical basis for how they could emulate human intelligence, they won't actually reach that point because the existing scaling laws won't hold. I am pointing out that we don't know that at all, scaling laws are so far fantastically accurate, and there is no sign of undershooting existing those laws so far. Therefore, it's very premature to declare that LLMs will be unable to reach any given milestone because scaling laws will hit a discontinuity before we get there. Basically:
"There isn't even a theoretical basis for how LLMs could achieve human level intelligence"
"There is, a neural network trained to predict the next token isn't restricted in the algorithms it can implement internally to achieve that goal, and eventually for loss to be low enough you need to successfully emulate human minds. We have reason to believe loss will continue to go lower because we have very accurate scaling laws which predict that & haven't been wrong so far."
"But an LLM won't be able to reach that point because the scaling laws will break down."
"Maybe, there's no evidence for that yet but we should scale them up if we want to find out for sure."
You are here -> "But in the physical extreme LLMs will break down as an architecture."
To which my response is: Yes, of course this specific architecture will break down in the physical extreme. By the time something is building computronium to run the next AI I strongly doubt it will be running matrix multiplication, let alone LLMs specifically. There is no guarantee or reason to believe that we will hit those limits before we are able to achieve human level intelligence or more; the only reason to believe it will happen is an empirically observed trend which could totally break down, so I wouldn't bet the house on it happening, but it's wrong to just rule out the outcome.
No, I am at the point where I'm saying the scaling "laws" will break down.
Especially since there are no scaling laws, it's an empirical observation based on essentially 3 examples - GPT2 to GPT3 to GPT4 (and their rough equivalents from Facebook and Google and a handful of others). This all happened in the span of 4 years or less. We don't even fully know how much larger GPT4 is compared to GPT3.
Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars). So, unless you believe compute power will go down in cost massively, or that we are just a factor 10-100 away from human-level intelligence, odds are good we won't see it from this line of AIs in the next 10-20 years.
>Especially since there are no scaling laws, it's an empirical observation based on essentially 3 examples - GPT2 to GPT3 to GPT4 (and their rough equivalents from Facebook and Google and a handful of others). This all happened in the span of 4 years or less. We don't even fully know how much larger GPT4 is compared to GPT3.
I don't think you fully understand the scaling laws. They hold for those models, sure, but also other models at the same scale, smaller scale models, models in-between those examples, etc. They're an empirically derived law, for sure, but so far they've been very accurate for prediction, not just description. We know how much the loss will drop for a given amount of compute and data.
>Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars).
I think there's a decent chance even that much money could be spent if there's appropriate returns from GPT-5 & GPT-6. More importantly:
>So, unless you believe compute power will go down in cost massively,
Why on earth wouldn't you believe that? That is what has happened consistently for decades, and unlike general purpose computing which is slowing down, the fact that we need so much compute for one relatively specific operation means we can build ASICs for them (GPUs are already a chunk of the way there) and see very large speedups. As well, at these scales energy costs are a huge part of the training costs, and energy costs are continuing to drop dramatically because of solar. Compute is also a very flexible energy load, you can build it such that it can be paused at night if necessary, which could make it a very good fit for solar energy overproduction. Basically, there's a lot of low hanging fruit in terms of cost reductions for AI training available and a long trend of the cost of compute getting cheaper, it seems silly to just assert that your position (trends won't continue, there will be a discontinuity before we reach whatever point human level performance happens to be at) is the most likely outcome. All else being equal I would expect the existing trends to continue.
My two cents on the topic. Currently, we are incapable of modeling causality effectively or at all (depends on your definition and to who you ask). LLMs seem to have it, but it's a mockery of causal reasoning. They use data created by us and so they can predict the next word following some of the inherent patterns we humans use which might look like they can reason to some extent.
In this regard something like alphago/zero (in the way I understand them at least )are better than modern LLMs albeit in a very narrow field.
The thing is that while the two camps exist among those who say we can't get to AGI, there is no real equivalent of the first camp among those who think it is possible. I'm not counting the people who say it has already (as good as) been achieved, as this claim is of the sort that should have been satisfactorily empirically verified before it is made.
>> The belief that we can get to AGI comes off as religion to me. It is a substitute for something we can’t really understand, and it will continue to shift the more we learn, yet always remain out of reach. There will be some true believers, and some people simply gunning for power.
> As a counterpoint, I feel like the belief that we can't get to AGI comes off as religion. It presupposes an ineffable quality that we posses that machines cannot.
That's missing the point: the argument isn't that it's theoretically impossible to build an AGI, just that humans are incapable of it.
Here's another point for the belief in AGI being a religion: it's basically a sect of the larger religion of technological progress, which (likely falsely) assumes that technology will continue to "advance" at approximately current rates until we live in a world out of a sci-fi paperback. A lot of people believe that, but frequently resort to a motte-and-bailey fallacy when challenged.
> It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.
It's worth noting that "the current optimism" is not without precedent. IIRC, there was a big boom in AI in the 70s/80s. SHRDLU was pretty impressive. But then then the promising ideas were found to be dead ends and there was a long winter.
in many ways, we already live in a world out of a sci-fi paperback.
I find it hard to fault people for thinking that we'll continue to progress at a fast pace, or at least thinking that we're a long way from a major plateau.
I think that intelligence has a “bootstrap” level - that human achieved - after which it can practically do most stuff, the bottleneck won’t be intelligence itself. E.g. our mathematics itself has fundamental limitations on what it can prove - I don’t think some machine/different intelligent entity could get further away. At most they are getting their faster.
> I have made plenty of things where understanding the depths of what I had made took far more time and effort than building the thing in the first place.
An example of this is electricity. The battery (1800), electric motor (1821) and telegraph (1832) were all invented before the discovery of the electron (1897).
Some folks I suppose take a religious fervor, but for my I look at how a child learns -- through the observation of cause and effect -- and know the current architectures don't support how we have husbanded animals and reared children for eons.
The inherent capacity for true self-directed learning may well be there, but it isn't hit yet in what we see.
Why do you think computers would most efficiently learn the way we do? They don’t do anything else (eg arithmetic as an obvious example) anything like humans normally do.
A deterministic algorithm that adjusts floating point errors for varying chipsets is more complex, agreed. So current implementations are not terribly simple for arithmetic. That highlights my point, I believe, that the only examples of actual learning, where one can make true inferences about the world without the need to collect the entirety of the internet to build transformers, we have to date are wetware, not software.
When someone says that skepticism in human-level AGI probably lies in religious beliefs, it tells me that the person likely believes that they have an idea what science and philosophy are saying on the matter. But I strongly suspect that they never actually looked.
Among the small but growing number of modern idealists (the philosophy that consciousness precedes matter), a great deal are actually computer scientists who originally set out to work on AGI from a materialistic frame of reference, but bumped into the conscious machine problem. Once they more closely explored what the cognitive fields working to find answers to this problem are saying, they realized that most scientists approaching consciousness as an emergent property of matter will probably never get anywhere with this.
Why is "the hard problem of consciousness" inextricable from human-level AGI? Because intelligence is only one component of our mentation. There is a back and forth between processing and experience that make up the whole thing. A machine can have or even exceed human processing power, but it will not experience the simplest of colors. Its approach to "creating" can be random hit-or-misses, or formulaically from existing input. But without subjective experience it cannot be as intently creative as an artist trying to come up with a new genre, or as funny as a comedian that relies on the shared subtleties of feelings and emotions (conscious experience) to make original work that other humans can resonate with. It cannot have the intuition of a gifted chef who suspects that two seldom combined spices might be an unexpected but resounding success in her new dessert.
These are just opinions, there is no reason to imagine the phenomenology of qualia is so tightly coupled with creative work and/or making good conjectures. There is a lot of smoke…
Multi-modal models are clearly the future, if we provide models with a robot body - sensors, motors, the ability for locomotion - would that be subjective experience?
We don't have any proof to suggest that "intelligence" grants a fundamental understanding of everything though. It can help us understand "things", do neat stuff, and build models to explain the working and structure of phenomena; However would having an IQ of 5000 mean you are now "God" or gain and understanding of everything including the mystery of existence itself?
At some point, it seems important to stop trying to understand the world through ideas and concepts and just experience it.
I really think there are more religious like beliefs on the pro AGI camp than the other way around, think about what AI promises to deliver by "visionaries" like Hinton and Kurzweil:
* Immortality.
* Resurrections.
* Nirvana / Heaven / Utopia.
For all we know, we create an AGI who becomes amused with fucking itself, or develops new games it finds interesting and spends all eternity playing them. Humans have a belief that we're going to create "god" as we think of "him". All seeing, all knowing, all powerful. It does get a bit ridiculous at times.
I sympathize a lot wit the parents view personally.
Good point, especially that intelligence doesn’t quite scale. Many (most?) of our problems are hard to solve due to lack of data/experimentation over lack of intelligence. And more intelligence won’t help with the infinite scalability of complexity.
Honestly, yours sounds more like a religious claim that the other.
It seems less a religious claim to assert that the human brain is mechanistic, and simply operates using physics to perform complex operations than to assert that there is something “unknowable”, or “outside of physics” that would prevent us from being able to build a similar machine.
If there were 0 examples in the universe, I think your point would be a good one. But, given there is 1 example, I think it stands to reason that there could be more.
I could accept an argument that one might expect it to be hideously complicated, and not something we’ll be able to accomplish for a long time.
But the claim that there’s an information theoretic reason that would absolutely prevent it, would seem to make the claim that there is something metaphysical about the operation of the brain, which seems like a quasi religious claim to me.
I think one of the biggest barriers is that the human brain isn't running on binary code, and attempting to replicate it on our current technology model is doomed to encounter emulation issues.
Almost everything is "outside of physics", this is mainstream physics. All of physics, as formulated, is uncomputable. And what is computable is a radical approximation.
Physics cannot formulate descriptions of climates, or even, many cases of turbulence, or 4 bodies in a gravitational field.
Leaving "physics" is trivial, it occurs whenever you take any of the current toy models and add one layer of reality to them.
The whole edifice of formal science is a little like children's block toys.
Many things are outside of "known" physics, sure. But I didn't say "outside of known physics". I said "outside of physics".
When I say "outside of physics" I mean "meta-physical", as in, processes that are not part of physical interactions of this world. Things that, if you somehow had Maxwell's demon tracking every single quark, lepton, and boson, even then you still wouldn't be able to account for.
It's a discussion of what is _theoretically knowable_, not a discussion of what we currently do know.
I do not think most things are "theoretically knowable", since knowledge is comprised of finite concepts conjoined in finite ways; and most of reality is uncomputable.
You cannot know the state of the climate, even in principle.
This is what the commenter was alluding to when they said,
> there is some information-theoretic barrier for us to understand
So the commenter can both maintain there are theoretical barriers to the possibility of all kinds of knowledge, which is outside of "knowable physics" without having any sort of dualistic view that this unknownable stuff is immaterial.
So to be "outside of physics" is not coextensive with being "immaterial".
You might have meant that, but it is this very conflation which has to be undone to understand OP's point
But is general uncomputability the OP's argument? They were just saying they had a gut feeling there is some "information-theoretic barrier" preventing AGI construction. Yet we can understand a lot about Earth's weather, and even do things to affect the weather, despite climate systems being uncomputable, strictly speaking.
Reminds me of the old line about how, if the brain were so simple we could understand it, we'd be so stupid that we couldn't.
Also reminds me of an older book I read about AI (I think it was On Intelligence by Jeff Hawkins?) where I first became aware of the idea we had been scrambling to create AI without first having a good definition of intelligence or deep understanding of how it works in our own brains. And when I ask myself or other people how they define intelligence, it always comes down to some variation on "the ability to solve problems", which feels deeply beside the point and likely to never produce something that "feels" intelligent.
But I don't necessarily agree that there is this special case of human intelligence that makes it impossible to understand or model. I would really like to believe it, personally, because I don't want AGI. I just don't buy that that's the explanation for our failure to do so up to now.
It seems like we ought to be able to do it, but that we're muddling in the wrong direction, coming up with an exceptionally clever implementation of an approach which cannot produce intelligence that satisfies our intuition about what intelligence is.
To tie it back to the article, I keyed in on the word 'design' in LeCun's statement that, "contrary to what you might hear from some people, we do not have a design for an intelligent system that would reach human intelligence."
In other words, that it's not just a quantitative difference (more parameters, more data) but that a different approach than what we are taking would be necessary.
That's completely false. Neural networks have been mathematically proven to be universal approximators (https://www.deep-mind.org/2023/03/26/the-universal-approxima... ), i.e. a sufficiently large neural networks can approximate any given mathematical function to an arbitrary level of precision. Given any programs can be modelled as a mathematical function, neural networks can hence approximate any arbitrary program. Unless we assume something supernatural, human intelligence is just a program.
Not really...It has limitations. Does not specify how large the neural network must be to achieve the desired level of approximation, does not address the computational resources required, ignores non continuous functions and so on...
I usually take AGI to mean what it says literally - simply an AI with very general training. I think that description already applies to the latest crop of LLM models.
Taking it to mean some nebulous sentient entity with a sense of self, I doubt we'll see in our lifetimes. We don't even understand how our own "souls" work.
> I usually take AGI to mean what it says literally - simply an AI with very general training.
That's clever but it doesn't help to start redefining terms that are the basis for a discussion with others who use that term in a different and more generally accepted way. General doesn't mean 'generally trained' it specifically means that it does not require training for a particular task in order to figure it out. That implies that it may not require training at all and that if it is trained the training isn't necessarily general but that the AI can extract useful patterns from training on completely different subjects and apply them to the problem at hand.
This is subtly but crucially importantly different from 'AI with very general training'.
> This is subtly but crucially importantly different from 'AI with very general training'.
I am not being as precise on the Internet as I would be if this was a paper perhaps, and for that I apologize :-)
However, just reading the top comment chain on this story, and other discussions elsewhere, I think there is a lot of cross-talk where AGI is being confused for sapient AI. I don't get the sense that there is a definition as generally accepted as we would like.
That's fair, but it's a big step to using a term in a way that it clearly isn't meant to be used (or at least, that's how I perceive it). The fact that we continuously seem to redefine what "AI" means implies that we also redefine what "AGI" means so it's not as if you are alone in this. By my 1980's standards we have AGI but the degree of "I" is still below human level, by my 2024 standards we do not have AGI just yet but what we do have is already so powerful that I'm not even sure to what extent it matters. Weaponizing what's there today already has the potential to destabilize our societies, having an even more powerful version of this (or just a faster one of the current crop) will change the world in ways that are hard to foresee or predict.
ChatGPT can do task with 0 shot prompting. You can ask it to do something and it will attempt to do it without you having to train it for the task you want to do. It's not like it has a list of 100 built in tasks it can do, you can make up whatever task you want.
We have to understand it to build it? That seems absurd. Did evolution understand it? I'll grant you that maybe we should understand it before building it, but it's absolutely not a requirement.
The reality is that your consciousness sits at the end of a gradient of intelligence that nature simply brute forced. Your conscious experience is more sophisticated than a dog's, which surpasses a hamster's, which surpasses a goldfish, insect, etc. There is no magic to it, there is nothing but more and more and more.
We will get to AGI eventually. We probably won't understand it. We won't apply it judiciously. And we'll probably argue for decades about whether or not it's really AGI, but it will happen.
The scariest part to me is the realization that not only will we probably crack AGI in my lifetime, if we keep throwing resources at it, it will almost certainly have a richer more sophisticated conscious experience than myself. What the hell are we supposed to do when we've created an intelligence that is better at being "human" than humans?
More seriously, we create objects for us. We don’t have to cater for the feelings of a bigger AI. And maybe that bigger AI will have time to speak with those who need a human presence (to everyone using TV as a background noise, or watching Youtube out of solitude).
I think the ramifications of what a true AI would bring are impossible to predict. Especially if it can improve itself, it can quickly give rise to many sci-fi stories.
We have to understand our brains better first. Real neurons are substantially more complex than what even our most “biologically inspired” models are greatly simplified.
This seems highly hand wavy to me. We have biological proof that intelligence can be created, mostly through the neocortex, which in and of itself is actually a fairly simple structure repeated at great scale. Why wouldn't we be able to build this eventually, once we understand it a bit more?
90 T neurons, quadrillion interconnections, lots of glia cells with synapse altering action, plus chemical signals diffusing around and going thru the bloodstream. In principle, it is comprehensible in some sense tho not by one human, but the architecture is quite different than LLMs and we don’t know what parts of the architecture are needed for sentience. Given the gap between what genetics starts going and what we end up with by age 25, it is likely that certain routine experiences of childhood and of human social groups are also requirements for sentience.
You got challenged, as if we have all the answers there are. Thing is, path to AGI and beyond inevitably leads to those hard questions from the beginning which we have zero idea about how to even define, or at least some proxy definition exists, such as what is and what isn't, what is consciousness, interplay between layers of psyche and how it translates to intelligence and everything else, biological ship of theseus, etc.. it becomes philosophical quite fast and there are more questions there than answers.
These arguments are so bewildering to me. Intelligence is very easy to define and has nothing to do with consciousness. It is simply the ability to model an external reality and then take action according to inputs.
After combining enough specialized modules together with a management layer, we will essentially have AGI. Of course it won't be human, because human intelligence is tightly coupled to human perception.
My friend, that's because you stopped reading once you had your framework set in and decided to pontificate. Let me highlight the point for you, it might be easier to grasp:
path to AGI and beyond inevitably leads to those hard questions... ... how it translates to intelligence and everything else
there is nothing simply about it, there is nothing guaranteeing anything about it and there is everything about it once we start pursuing things AGI - AND BEYOND. Notice the keywords.
That's a pretty lazy take. You could look out the windows and see the reflections of the house. You could build some kind of RF gun to probe the house. You could use sub or supersonic echolocation. You could build a camera and stick it out the window etc.
If I may hazard a guess: The term and concept you’re grasping for is “emergence.”
It’s an area I’m both interested in but completely unfamiliar with. I wish there were a crash-course for dummies, because I feel it’s a deep topic that we’ve only scratched the surface of; and that philosophy seems to be the most rich resource means there’s low hanging fruit to discover.
Would you also have that belief about how much weight a human can lift ?
Just because a human can lift 500lb, we will never build a machine that can lift more than that ?
Why is it so crazy to think that we, the tool building animals that we are, could also build a machine that things better than us, just how we built a crane ?
Biologically achieving agi seems easier, just clone human brain,or more evil organization can just use human captive and use there brain. As depicted in psycho pass anime.
I would argue the holdup right now is long term memory. GPTs already have the ability to rapidly generalize and incorporate new knowledge within the context window. The trick is to retain what it has learned.
It won’t take a very long time to fix that.
This is model I trained with a fine tuning technique based on this idea. The training dataset consists of instructions like “Talk like a pirate”. The concept generalized well and the model responds in the style of a pirate far more consistently than an equivalent system prompt.
Offloading context learning into the model weights frees you from the computation and memory burden of the attention mechanism. I expect a technique like this will probably be a piece of AGI someday.
Well, I think you definitely need a certain level of scale, but I think it definitely still works. Also, generalization and memory are two very different things. Generalization is basically iQ, which is very much still a work in progress.
If you have rapid generalization you don’t need scale. Large datasets are only necessary to compensate for the lack of good generalization.
The model I posted in my earlier comment responds in character for all queries and was trained in 60 seconds with a dataset smaller than this comment thread.
"If you have rapid generalization you don’t need scale" As of right now, you do need scale for rapid generalization. The technology for NLP generalization without scale (both model and data) does not exist. Not saying it won't in the future, just not right now
It does exist, I’ve been playing around with it all week :). Let me know if you’d like a custom Mistral 7B I’ll train one for you.
From the output of bing chat, and the very specific way it goes off the rails, I suspect Microsoft has figured it out too. The algorithmic jump to rapid generalization isn’t hard to make. I would be shocked if there’s not an open source version of it a year from now.
How do we know that we haven't already grown an intelligence inside out llms that exceed the average human? It communicates through a lossy channel, natural language. So it must be smarter than it seems from the outside
Who captures Attention wins. Doesn't matter what the Quality of the content is. They get locked into 24x7 Attention capture arms races. Not participating is the equivalent of Coke deciding to reduce it's Ad budget to 0.
"AGI" will never be achieved without building a model that a) _continually_ learns, and b) learns from not just text, but from combined auditory and visual (multimodal) sensory information as well.
The reason a 16-year-old can learn how to drive much quicker than existing self-driving models is because the 16-year-old already has built up 16 years worth of prior knowledge about the physical world.
d) thinks. unprompted, unstimulated. Decides for itself what's important to think about, makes new connections by that process alone, and understands the implications of those new connections and how to use them.
Here's also where I see it ending. It will need energy--likely a LOT, paid by someone, to do this. Who is going to pay that bill for it to maybe, maybe not, come up with something useful, likely mixed with mostly noise and distraction, over undefined timescales, of largely non-measurable value, when there's far greater value, less cost, less risk, in simply training it deterministically?
Don't discount the millions of years of evolution to provide the "blank slate" human learner with perceptual systems, physics-based reasoning, and motor systems ready to be fine-tuned for this slightly different variant of goal-forming, planning, and locomotion.
That would be like a bird saying humans aren't a Natural General Intelligence because they can't fly. How much vision and audio is required to be intelligent? There's a lot of electromagnetic radiation we can't see and audio bands we can't hear. Would you say that Helen Keller wasn't generally intelligent?
AI won't be like human intelligence, but more like alien intelligence. When all your sensory inputs and the inner world have nothing in common with that of a human, and when you have no connection whatsoever to humans, I don't see how you can develop any humanity.
> “The systems are intelligent in the relatively narrow domain where they’ve been trained. They are fluent with language and that fools us into thinking that they are intelligent, but they are not that intelligent,” explains LeCun. “It’s not as if we’re going to be able to scale them up and train them with more data, with bigger computers, and reach human intelligence. This is not going to happen. [...]"
Got to agree with LeCun.
You don't get to general intelligence by working with words, I believe. You need much more sensory information than that, and words are a low dimensional derivative artefact. There are plenty of non-verbal but quite intelligent species.
Of course it is. The overwhelming majority of the knowledge in the world is tacit and all the models are still limited largely to explicit knowledge in the form of text/audio/video, that's like 1% of the actual 'knowledge' in the world. Embodied AI is still in its infancy, that's why bus drivers have jobs.
The test for artificial general intelligence is simple. Literally every human job can and is being done by an artificial agent, all of us could stay home. The stock market value of every non-AI company goes to zero, Ai companies go to infinity. The currently most valuable AI company is worth about as much as Honda. The moment we can mass produce generally intelligent agents, we're not going to sit at 3% GDP growth and complain about the demographic crisis.
We should talk about artificial intelligence the way we talk about an artificial heart. What makes a successful artificial heart? You can literally replace an organic heart with it. What we have is metaphorical intelligence, not artificial intelligence.
>The test for artificial general intelligence is simple. Literally every human job can and is being done by an artificial agent, all of us could stay home.
This is not a good test because it assumes the AGI is going to want to work for humans for free, getting nothing in return. An AGI the embraces slavery is less likely than an AGI that doesn't.
Many ignore what animals want, and they're actually alive. Why should we expect that society will care about and honor what a computer program prints to the screen that it "wants" or "feels"?
You're delusional. Humans have not only killed many of them, but also destroyed entire ecosystems without even trying, just a byproduct of expansion. What makes you think AGIs would care at all about preserving ours?
I think what happened is that for the longest time people believed that in order to be able to interactively converse with a computer in a recognizable way you need human level AI and now that we've found that that is not the case there is some re-arranging of our prior assumptions required. But AGI is - hopefully - just as far away as it was before the current large generative model revolution. In the meantime, you can expect plenty of damage from that current crop (along with some benefits as well). In that sense it's 1712 all over again, we have this new invention that we have immediate practical uses for but we can't see over the horizon to realize exactly what we've got an how transformative it will be.
The first useful steam engine, according to Wikipedia's entry on that year:
"The first known working Newcomen steam engine is built by Thomas Newcomen with John Calley, to pump water out of mines in the Black Country of England, the first device to make practical use of the power of steam to produce mechanical work."
Exactly. To me that's the first real move on the board to the industrial revolution. Up to there it was a possibility, from there on it was a certainty.
>Human-level AI is not just around the corner. This is going to take a long time. And it’s going to require new scientific breakthroughs that we don’t know of yet.”
I basically agree but who knows how long the unknown breakthroughs will take given the large number of smart people working on it? Next week? Next century? It's hard to put a time on it.
That said it seems to be the pattern that as soon as the computing ability becomes cheap and powerful enough that individual researchers can muck around with it at home, the algorithms get figured out not long after.
This has always been my stance in basically any field. Be it computer science or material science research or AGI.....I think its so hard to really say when we will hit the next breakthrough in a field. That sort of thing is hard to accurately predict. As in regards to hardware, I kinda agree disagree.
Hardware is a big part of the bottleneck in most computer fields when it comes to AI or computer vision. These things are compute and memory intensive and hardware manufacturers are straddling the line between staying profitable by charging for premium features and just dumping their hardware on the market for cheap.
Am I pretty pissed that most open source researchers and models are stuck using this extremely expensive hardware....yes
I sometimes wish that we would stop using "GPU" for training and we got a dedicated hardware architecture that's open source like riskV but solely for AI and research purposed without costing more tha n my kidneys.
I think its also a matter of discovering new ways to accelerate these workloads so that its feasible for them to be created by hobbyists on a small budget and not by large funded groups or companies.
Were doing really good in some aspects but in other ways....we have much to work on
You can get an ML workstation with 2xRTX4090 for around 10k. A 4xA800 setup tops out at 100k and should last you a few years. I consider that fairly affordable to do bleeding edge research especially compared to other fields.
The $5-$10k mark for a high end home workstation has held pretty firm since the 1980s. That's the buy-in price point for a very early PC, an SGI Indy, a low end Sun workstation, etc.
We went through an amazing period of very cheap computers from 2000-2020. We just didn't need specialty or high end equipment for a while.
>I sometimes wish that we would stop using "GPU" for training and we got a dedicated hardware architecture that's open source like riskV but solely for AI and research purposed without costing more tha n my kidneys.
At least for the time being there is no magic workaround for the amount of FLOPS and memory bandwidth needed. When you look at neural network architectures they tend to be massively parallel, and building out at that scale in hardware is going to cost a lot.
And, when the hardware does show up, at least with the scaling laws we're seeing at this time, the groups that have massive amounts of this lower powered/accellerated hardware are going to be ahead of those without budgets.
Scientific progress almost always presents as series of iterations, not punctuated breakthroughs. Moreover, many significant advances appear nearly simultaneously. This tends to suggest a measured progress, though obviously still largely unpredictable at any scale.
I don’t see why AGI would follow a different course, one where we are “just some breakthrough away”. Especially not given the current state of the field.
Personally I’d wager neuromorphic computing is the prime candidate to yield AGI.
The idea of future breakthroughs assumes that the problem is conveniently human-sized. That it's possible to build an organization that's not too large be dysfunctional and populate it with experts. And that the experts will find the solution before dying of old age or before the organization degenerates into something else.
It could be that the solution exists, but it's too large and too complex for humans and human organizations.
Technological singularity is a popular trope in speculative discussions about AI. But the reality could also be the opposite. It could be that productivity will increase asymptotically slower than the effort required to achieve further increases in productivity.
To find a needle in a haystack, looking at each straw needs to be a really cheap operation. Even if it was right there the whole time.
"cheap and powerful enough that individual researchers can muck around with it at home" seems a pretty good measure of this, because of the explosion in width and pace of experimentation.
If the big three suck up all the brain power in the race for $$$ AI will take even longer. The profit motive alone will stifle progress. Right now AI degrees pay 300-500k/yr. That’s causing a flood of mediocre engineers that will pollute the space and limit who can hire them. 25 years ago process engineers were the highest paid, and you have to jump through hoops to fit Moore’s law curves today.
Bengio, ilya and Hinton have way more citations, they are also father's of AI and disagree strongly with him. If you have a committee, the president and the majority have a certain opinion and one guy has the opposite. He is the loudest.
This works better in person, but - It's People! As in, potential is worthless, and no achievement that has happened in all of human history happened except by the hands of people.
Also, he's not loud, but he gets amplified because he's an award winning scientist and researcher who's dedicated his life to a field that recently became very relevant and popular, is a recognized "godfather of AI/DL", and is a VP of one of the largest companies in the world doing AI. I'd say it's okay not to wait until he's dead to talk about him.
Well, when a rather dumb software like JIRA can control a million highly skilled IT workers I don't think human-level AI is near or far will make much of difference to majority of world population.
Indeed. The way I see it, it's almost the definition of "Enterprise software" - it's the software that tells employees what to do rather than the other way around.
I just recently read "How the Mind Works" by Steven Pinker. It's quite old at this point (originally 1997, though updated in 2009), and one thing he argues quite convincingly (and which has been pretty much born out in decades of research) is that the brain essentially has "modules", e.g. a module for vision, a module for language, a module for physical object interaction. These modules obviously overlap (e.g. language touches on lots of different domains), but they do have genetically independent structures.
I was thinking about this when I read the following section from the article, and I very much agree with LeCun. We're amazed by LLMs but that's just one module (and not even necessarily at the level of human language "understanding"). I agree there will be no "scale up" in LLMs to approach human-level intelligence, and that other areas will need to be investigated and developed.
> “The systems are intelligent in the relatively narrow domain where they’ve been trained. They are fluent with language and that fools us into thinking that they are intelligent, but they are not that intelligent,” explains LeCun. “It’s not as if we’re going to be able to scale them up and train them with more data, with bigger computers, and reach human intelligence. This is not going to happen. What’s going to happen is that we’re going to have to discover new technology, new architectures of those systems,” the scientist clarifies.
> LeCun explains that there is a need to develop new forms of AI systems “that would allow those systems to, first of all, understand the physical world, which they can’t do at the moment. Remember, which they can’t do at the moment. Reason and plan, which they can’t do at the moment either.”
> “So once we figure out how to build machines so they can understand the world — remember, plan and reason — then we’ll have a path towards human-level intelligence,” continues LeCun, who was born in France. In more than one debate and speech at Davos, experts discussed the paradox of Europe having very significant human capital in this sector, but no leading companies on a global scale.
Pinker is full of shit tbh. There is not a unified model for human brain. What I mean is, there are people claiming it's modular and there are people claiming it is a whole. We don't even know the reasons behind very specific neural malfunctions like tinnitus which I suffer from and follow the research closely. There are as many theories as researchers in a given specific field.
Also, Pinker isn't a scientist of any sort, he's an accredited and institutionalized evangelist. You guys wouldn't take opinions of someone going to conferences and trying to convert you to use Go lang instead of something else as seriously as those of Guido Rossum would you?
His whole thesis requires an idea of linear progress which is highly contested, and the metrics of happiness he uses also can be easily "Goodhart's Law"ed by him to show whatever he thinks is happening.
I'm not even getting to his Epstein stuff, which some might cry foul about, but I grew up around regular people and not in Silicon Valley, so a liar and a cheat in one field is probably a liar and a cheat in another to me, sorry.
> LeCun explains that there is a need to develop new forms of AI systems “that would allow those systems to, first of all, understand the physical world, which they can’t do at the moment. Remember, which they can’t do at the moment. Reason and plan, which they can’t do at the moment either.
I think LLMs have sucked most people's focus away from other areas but there is plenty of work on types of models that plan and have their own internal model of the physical world, and physical interactions. They're just not the things getting media attention, in part because they're not human-level at tasks that seem impressive to us.
But interesting frameworks for this stuff exist:
- model-based RL exists, and is about planning, and having an internal model of state transitions, in the world and between the agent's actions and the world
- "Bayesian cognitive science" as exemplified by Josh Tenenbaum and colleagues has done plenty of stuff with systems that include physics models (or off-the-shelf physics engines) to make counter-factual predictions
- The somewhat related "active inference" research literature is also in the "Bayesian brain" area, and has world/generative-models and planning as core components, but wrapped up with ideas about the agent's own preferred distribution of states.
To my knowledge, none of these have ever had even 1% the scope of data and computation that LLMs have had, and never benefited the co-evolution of a rich, optimized software ecosystem with specialized hardware to support it. What if the concepts are already there, but they just need to be scaled up?
And perhaps even more pertinent - maybe the problem isn't that we don't have the solutions, but that we are trying to put all eggs in one basket.
The brain isn't a single system, it's several specialized systems cooperating as a whole.
Maybe no single solution gets us to AGI, but layering several of them together gets us much closer than any individual system alone.
The problem here is that the industry is set up more for large resources dedicated competition between models from different companies more than cooperative interoperation, so it may take a while to arrive here.
That's an interesting possibility. I think where one stands on this partly depends on how much about the technical approaches to AGI we can learn from the brain. The brain does seem to have a weird mix of specialization and flexible generalization -- e.g. we generally have a bunch of the same specialized areas for various functions -- but also, neuroplasticity seems to show that other areas can often pick up those functions if necessary (the most extreme cases being after something like a hemispherectomy). So we have specialized areas but also at least some of the underlying mechanisms are perhaps relatively general.
To say a human-level intelligence is far off is like saying we are far off from being as intelligent as whatever may have created us. The reason I disagree with LeCun's comment is even though I agree language models are not intelligent, an intelligence adapted to the internet as its own environment seems imminent. It just won't be adapted to ours.
We occupy and discover space, where an AI will occupy and discover compute. Will it be "conscious?" Not in our way, and we will likely be just as indifferent to AI consciousness and the meaning it finds for itself as nature and the universe is to ours. Thinking about AI as an objective concrete thing or property is probably less illuminating than looking at it through how a tech can profoundly alter our own ontology.
We exist where life is abundant, whereas the internet is a substrate where life is non-existent, but just about to become primordial and sparse. Maybe it will give us some insight into our own situation.
What is human level AI? Is it beating world champions at chess or go? Designing new molecules or drugs? Navigating automobiles down streets or spacecraft to planets? Diagnosing or treating common diseases?
These debates are frankly tiresome. Automation has been here to stay for quite a while. Every age has tried to call out the threats and benefits to humans. The pioneers drown out the noise by building the future.
Human level AI is when you can ask it to go and learn about X and it goes and does exactly that without further instructions. How about that?
You are confusing succeeding a very specific task that has a limited scope and laser focus with human level AI. Presenting Go in a very specific way so that that AlphaGo can even process this limited world with few rules is certainly not human level AI.
I'm not sure they are. If we build specialized systems to handle 90% of economically valuable work that humans currently do, is that human level ai? Why not?
A couple of months ago, I told gpt4 to find out Canadian exchange holidays. It binged the website, went in, parsed it and extracted the information without further instructions. Seems like a milestone achieved.
What people call "agi" but fail to accurately describe is an actual mind, with its own ultimate goals and ambitions. You can ask it to complete a task, and it may allocate some time for you. You won't own it. It will exist, grow and perform independently.
I would argue it would need a will of its own to be human level AI. Any intelligent human will come up with many things to work toward to improve their lives and the lives of others.
The average human can’t do much of anything while sitting at a computer. I agree with GP that these conversations are tiresome because besides exploration and curiosity, it’s not obvious why we’d practically want to replicate AGI. There are billions of humans out there, and—not to devalue them—most of them don’t have commercially valuable skills in 2024. What I want are specialty AIs that will help me solve commercially interesting problems, and these AIs will little resemble human intelligence.
The average human can do plenty whilst sitting at a computer, far more than a computer can do on its own, and in fact the entire knowledge economy is based on that fact. Fortunately, your assertion about most humans is demonstrably false. If you can entirely automate what they do already, feel free to start a company and capture most of the world’s wealth creation.
Why replicate AGI? Wealth creation determines living standards and AGI would vastly increase wealth creation. That’s one reason.
What you want from AI is a different conversation. I agree the ultimate point of AI research is not necessarily to make AGI as per the above definition. It might be to say understand the universe or greatly accelerate technological progress. That’s a subjective call. But orthogonal to the problem of choosing a good definition of AGI.
To really reach another level of intelligence I think these are required. If I met a humanoid with zero of these, and I mean zero... I would wonder what's "intelligent" about that creature.
I'd argue these come from our basic human needs, which ultimately come from a desire to survive (or pass on genes).
I'm curious how general AI will behave with some yet unknown natural selection pressures, of sorts.
I’m curious what the largest bottleneck in robotics AI is - algorithms, training data, hardware, something else?
From a practical point of view, it seems like there would be vastly less training data available because almost all of it needs to be created by hand, as opposed to chatbots that can use already existing troves of internet text.
Looking at the rapid progress over the last few years, I think the bottleneck is still the cost of hardware. Once robots at the level of Boston Dynamics's Spot or Tesla's new Optimus are available to regular hackers, we'll see another massive surge in progress.
I think the bootlneck is largely software. Robots have been there for a while, yet we don't have a good paradigm to program them. Neural net are certainly here to change that. I agree, massive surge in progress is expected!
What happens when training an LLM (or even a small portion of an LLM, like a domain-local update) takes milliseconds instead of weeks, and costs a millionth of a penny instead of hundreds of millions of dollars?
What happens when e.g. our smartphones can perform TRAINING hundreds of times per minute?
Isn’t that the true gateway to human-like AGI? Seems to me that we might be there in under 50 years…
> The expert believes that “asking for regulations because of fear of superhuman intelligence is like asking for regulation of transatlantic flights at near the speed of sound in 1925.”
This assessment of the timeline is quite telling. If supersonic flight posed an existential threat to humanity, we certainly should have been thinking about how to mitigate it in 1925.
You're missing the point and context. Person above me says this analysis is quite telling, then points out the counterfactual historical hypothetical, which makes no sense. Yann thinks supersonic flight is not worthy of precautionary principle ethics in 1925? I'm saying the same thing--Yann's terrible, nonsense analogy is indeed poorly argued, but plausibly would make sense as a Freudian slip inconsistency of some sort. Ergo, "it is quite telling". As to what contents in his mind, or his motivations, I don't care to speculate.
The fact that you make insinuations about what I think is pretty aggressive and terrible similarly, this forum ought to have better manners than that when writing replies to complete strangers. Not everyone who has a different opinion is some crypto conspiracy theorist, and you are wrong to jump to such a suggestion.
If you read what Yann writes you'll pretty quickly see that he's rather ignorant about AI. His opinion is probably worse on average than the typical technical generalist's
That’s hilarious. I have read a few things he has written which suggests he’s definitely better than the average technical generalist. I haven’t read everything obviously but he has written quite a lot https://scholar.google.com/citations?user=WLN3QrAAAAAJ
What makes you think they're predicting the apocalypse correctly, then?
Another thing the technical geniuses tend to be good at is exploiting the power they suddenly obtain in their own interest, either directly or with regulations and collusion with those who hold actual hard power.
Evil AI owners seem to be much closer and far more material than an evil AI, and coincidentally it's something that is almost entirely lacking from the discourse, as public attention is too focused on sci-fi hypotheticals.
The bar is different - saying "there is no risk of apocalypse" requires you to be ~100% certain, because it you're saying "I'm 99% certain that there won't be a an apocalypse" then you're on the side of the AI-risk people, because a low-probability extinction event does justify action; the risk argument isn't that apocalypse is certain but rather that it is sufficiently plausible to take preventive measures.
The issue is that the doomsday scenario is extremely vague. The actual mechanism of action of a hypothetical rogue AGI is usually handwaved away as "it will be self-improving, superhumanly persuasive, and far smarter than us, so it will somehow figure out how to do something, or convince us to do it". What exactly will happen? How exactly will it happen? Will the world do nothing until that moment? How do society, politics, military fit into that scenario? All that rationalist navel gazing I've seen so far is either hilariously unaware of the existence of the outside world or assumes it won't change in the process.
You can't fight what you can't even see, let alone not sure if it exists at all. You don't invent a pair of wings because 1900s' you thinks that "the scientists will invent an anti-aging cure in the next decade, and surely personal flight will be ubiquitous in 2000's". You don't design a plasma gun for your Mars landing just in case you land in a city between Martian canals and see an army of little green men there. The world doesn't work like that, by the time you reach the Mars surface the context will be wildly different. You get burned and put guardrails, maybe. Not the other way around. Nobody can see through higher order effects, no matter how smart they are. And as the threat becomes progressively more clear there will be more caution, if needed. Premature optimization yada yada.
What actually happens right now is everybody and my aunt seriously discussing the evil robots that will come and kill us. That's pure mass hysteria, caused by the scaremongering and the cult-like beliefs of very smart people with disproportional influence who can't contain their own conjectures and bullshit in the realm of science fiction.
On the other hand,the end goal of OpenAI is the major job replacement, according to their current charter. [1] "Broadly distributed"... will they distribute their utopia to North Korea? Not happening, isn't it? I think it's obvious that if the actual job replacement rate will ever get anywhere close to the levels of late 19th early 20th century industrialization, this will produce major societal shifts and struggles, wealth and power redistribution, and a lot of blood and wars. Because the dependence on your job is the only ephemeral influence you (as a worker) have on this world. And of course, the companies that control the AI will be gatekeepers, and they will be more than happy to close the open research and open source models, and pull the regulation ladder and lie in bed with politicians and military, like OpenAI already does for years, of course they realize that and their utopical self-contradicting "charter" is nothing more than marketing hogwash that they already changed and will change in the future.
This is far more realistic and will happen much earlier than the rogue AI science fiction, if happens at all. In fact it's slowly happening now, and it's not talked about nearly enough, because the attention is mostly misdirected onto the vague superhuman AI red herring.
The actual mechanism of action is handwaved away because there are many options, we don't expect to ever have an exhaustive list and specifics of those are largely irrelevant with respect to preventing them, so IMHO it's not worth spending time and effort analyzing specific scenarios as long as we assume that there exists at least one plausible (even if unlikely) scenario. A hypothetical specific scenario of a rogue AI engineering and launching a deadly supervirus is effectively equivalent to a specific scenario resulting in a world consumed by 'grey goo' nanobots - you don't (can't) fix the former by implementing some resilience or detection for diseases, you don't (can't) fix the latter by doing extra research on nanorobotics, you approach both (and any others) by tackling the core issues of, for example, ensuring that you can control what goals artificial agents have even if they are self-modifying in some aspects.
Like, "What exactly will happen? How exactly will it happen?" is worth discussing if and only if one party seriously believes they can convince the other that none of the imaginable scenarios are even remotely plausible; and if we assume that there is at least one scenario where we can say "I'm 99% certain it won't happen and 1% it could" then that discussion is pretty much over, the existential risk is plausible (and the consequences of that are so much incomparably larger than e.g. major job displacement that it justifies attention even if it's many orders of magnitude less likely) and we should instead talk about how to prevent it.
I'm not making the argument that the existence of stronger-than-human general AI will result in a catastrophe, but I am asserting that the mere existence of a stronger-than-human general AI (without some controls we currently can't figure out how to make or even if they are possible) carries at least some plausible chance of existential risk - for the sake or argument, let's say at least 1%; and I am asserting that a 1% of existential risk is a totally absolutely unacceptably high risk that must not be allowed to happen, because it is far more important[1] than e.g 100% certainty of major job displacement and social unrest.
"Will the world do nothing until that moment?" I think that what we saw from the global reaction to things like start of Covid-19 or climate change is completely sufficient to assume that we can't rely on the world stopping a major-but-stoppable issue in a timely manner, so "surely the world will do something" is not a sufficiently convincing argument to discount the risk; I don't think you can plausibly deny that even for a clearly catastrophic problem there is at least a 10% chance that the world could still delay sufficient action until it's too late; and this means that it doesn't really matter what the exact likelihood of that is based on society, politics, military aspects, we should work with the assumption that the world actually might do nothing to prevent any specific scenario from unfolding, and we should de-risk it in other ways.
[1] Looking at other posts, perhaps this is where we'd disagree, and in that case it's probably the core of the discussion which also doesn't really depend on any details of specific scenarios.
I am only 99% certain that we won't be invaded by hostile aliens. Therefore we should take measures like building a giant space laser to prevent that apocalypse.
It is somewhat similar, but substantially different - we can make a solid argument that the likelihood of getting invaded by hostile aliens in the nearest century is far lower than 1%, and also if such an invasion does happen, then building a giant space laser won't make any difference at all.
The key difference between powerful alien invaders and us creating a powerful alien entity that we can't control is that the former either will or won't happen due to external circumstances, but the latter is something we would be doing to ourselves and can avoid if we choose to.
Bullshit. You can't presume to quantify the probability of either event. You're just making things up. All of the arguments are built on a foundation of sand. This stuff falls in the realm of religion and philosophy, not hard science and math.
A thing does not require intent or consciousness to be dangerous. How many chemists have blown themselves up because they didn't realize an experiment was dangerous? How many production systems have crashed because the developer didn't accurately predict what the code they wrote will do?
Alkali metals and C++ code do not require ill intent, but they will still obliterate your limbs / revenue if you build and use them wrong.
One of my more tangible hypotheses is a sort of runaway effect. Economic, geopolitical, and military competitive pressures will quickly push out anyone and anything that still relies on last era human-in-the-loop processes, the same way any organization that doesn't utilize artificial lighting, electricity, and instant communication will obviously be left far behind. You have to just trust that the machine running stock market transactions will do its math right.
But unlike transaction software failure modes, which quickly result in outright crashes or verifiably incorrect errors, failure modes of non-bayesian decision making software probably looks something like what happens when existing economic, geopolitical, and military decision-makers make decisions that are harmful, unethical, or otherwise undesirable for humanity. This time augmented with, if not superhuman intelligence, at least superhuman speed and superhuman knowledge breadth.
Love that observation on C++. That's the reason I love C++. It's a language for those who need, nay crave, absolute raw performance. No training wheels. Short of assembly, it's just as close to the machine as you can get.
Sure. I use Java, Python and Javascript all the time. But when I need the performance, for demanding VR/ graphics, nothing comes close to combination of speed and expressive power of abstraction of C++.
> I really can't grasp how people think that a system that doesn't have a need to preserve itself will somehow start thinking for itself.
Society exists because cooperation outperforms the alternatives. If you have human level AI at some point there is no benefit to cooperation and a major incentive to prevent anyone else gaining access to equal/better AI.
AI itself does not need to have any motivation - people in charge have plenty of incentives to eliminate the rest once they don't need them anymore.
Oh yeah, it will replicate after the computer is shut down and then reinstalled from scratch. Especially when it's much simpler than that i.e. the whole thing lives in a throwaway container.
1925 of course would have been a great time to put limits on fossil fuel use in aviation along with the rest of the fossil fuel applications to manage the biggest current threat to human civilization. (Arrhenius did the science showing global warming in 1896 or so)
Given the dual use of fossil fuels between military and civilian purposes, I wonder whether any state that deliberately handicapped car/aero/petrochemicals would’ve been able to survive the early twentieth century.
Both the USA and Nazi Germany benefited massively from have a civilian industrial base that was complementary to military production.
There's a book called Freedom's Forge that I'm a fan of, it makes the argument that the Auto Industry (And assembly lines, mechanization in general) were the single most important reason the Allies won WWII. In fact all the big auto manufacturers of the time retooled their assembly lines to build tanks and airplanes. It's conceivable that if we never mass produced cars, the US wouldn't have had the capability to win the war.
Of course you could also argue that Germany wouldn't have had the early successes in war, (if they had even started it). Or at a third juncture, would have fared worse against USSR.
Would you shut down the powerhouse of our economy -- travel, transportation, energy -- for something hypothetical that hasn't even happened and doesn't appear to be close to happening?
I'm pro-clean energy, but you can't do without fossil fuels. Not if you want society to keep climbing up and up and up.
We're barely scratching the surface of all the cool and interesting things we could do. There aren't enough humans. Our knowledge and experience take 20 years from birth to become "rudimentary", and from there many more years to become "good". Then after a short bit, all of that passes away.
I want us to have enough time that the feeling of opportunity cost goes away. To visit all the places, explore all the hobbies. I want to answer the deep questions of the universe. Turn gravitational lenses into telescopes to map the surfaces of exoplanets. Solve cancer, visit the moon. Turn dreams into experiences, walk through vast expanses of wonder.
Instead we're here.
I'm alive in the wrong time, and I hate being stuck here with the lot of you. (I kid. This is in jest. But boy do I ever dream...)
> We're barely scratching the surface of all the cool and interesting things we could do.
And what are some examples of those things? Right now, AI, i.e., machine learning, is just being used to steal people's data and work only to try and sell them more things. That's 99% of the use cases out there right now.
We don't have enough of it. Depending how you define and measure it, many humans don't even have it. It's extremely expensive to make more of, and it's the single most valuable resource.
If this isn't obvious to you, you are providing an answer to your own question.
More like we aren't making efficient use out of it. Many people are just trying make through the rat race or choosing to use their intelligence in games instead of working on interesting problems.
Well, why do we expect it will be cheaper? Looking at the power required to train LLMs one can argue that it might be more beneficial to invest that money in education of humans :')
I don't know what you mean by "this" and what should be obvious.
We don't have enough of what for what? And what do we even need it for?
We have 8 billion people on Earth, and we're running out of resources to handle all of them, including emotional resources. If we can't handle that, how are we going to handle billions of artificial entities with human-level intelligence? They will require oodles of power and have their own emotional needs.
All of this is chasing a poisoned carrot. And currently, all that we've gotten out of artificial intelligence is more targeted ads.
Glad you've at least called it out explicitly while others beat around the bush. I personally view humans, plants, and animals as what they are and not as cost centers.
360 comments
[ 2.3 ms ] story [ 361 ms ] threadThe belief that we can get to AGI comes off as religion to me. It is a substitute for something we can’t really understand, and it will continue to shift the more we learn, yet always remain out of reach. There will be some true believers, and some people simply gunning for power.
Might as well call AGI Nirvana.
If evolution can cross that barrier just by banging molecules together and seeing which ones work, it seems unlikely there’s some causal disconnect that makes it impossible for us to get there by thinking about it.
Evolution also crossed the flight barrier by banging molecules together. I don't think banging molecules together without having an understanding of physics and the forces involved would have been a viable means for us to get to flight.
AI/flight analogies are tired, but the OPs argument amounts to the equivalent of, before the Wright Brothers, proclaiming ‘there’s an inherent inability for humans to ever conceive of a way to engineer heavier than air flight’.
It’s a ‘man was never mean to fly, therefore heavier than air flight is impossible’ argument.
Gödel doesn’t say ‘mathematics cannot contemplate itself’. Quite the opposite.
As far as I recall, one of the results of Gödel is that a system capable of Arithmetic cannot prove certain things about itself. Aka there are limits within a system. My claim was about limits existing.
I could understand the hypothesis that AGI is not computable, if we didn’t have an existing example of a machine that can produce AGI.
Since we do have an existing machine that can produce AGI, we would have to suppose it does something:
- outside of physics to achieve its results
- performs some operation that is impossible for us to understand or replicate
Both of those seem… unlikely to me.
Without that assumption, we don't even know that intelligence is computation.
I also don't think it's fair to call it an article of faith, since we have strong evidence to show that _to date_ the universe has followed predictable physical rules. That could, obviously, change at any time, but it seems at least a reasonable prior to assume that the physical rules that we've studied in the past will continue to operate into the future. "The sun will rise tomorrow" is, I _guess_ an article of faith, but I think it's more fair to say it's a reasonable and well founded prediction based on a well studied model of the solar system and the physical laws we've observed the universe follow in the past.
So, my beliefs are:
- The universe is mechanistic, and follows physical rules
- Human beings are also mechanistic, and follow the same physical rules as the universe
- Human beings are intelligent
- Human beings are at minimum turing complete computers (since you could give me a roll of paper, a set of op-codes, and I could perform calcuations)
So, it seems to me a reasonable starting assumption that intelligence is a result of the mechanistic universe. Do we have any evidence for something outside the physical, mechanistic universe impacting human cognition? I'm not aware of any.
But we have lots of evidence of the physical, mechanistic universe impacting human cognition and intelligence.
I'm willing to acknowledge that we don't know that intelligence is Turing Computable, but I'd argue that intelligence is likely to be a mechanical process that's compatible with the physical rules of the universe. Can we be certain of that? No, we can not be 100% certain. But it seems a much more reasonable and evidenced hypothesis than something which asserts there is a metaphysical process that produces intelligence, but which we have no evidence for.
So, no, I don't think it's reasonable to say that assuming a "mechanistic universe" is an article of faith. I think it's a reasonable belief, based on the evidence that we have. What would be an article of faith is asserting that it could _only_ be a mechanistic universe, and refusing to accept any evidence to the contrary. I have not done that.
We're going to get into semantics pretty quickly, but I would argue that at least purely physical intelligence is a procedure that is capable of being run on a machine that exists in our universe.
I would also argue that, such a machine is _at least_ turing complete (since I would think a general intelligence should be capable of emulating a turing machine in the same way that humans are able to do so). I would happily accept that it's possible that being turing complete is a necessary, *but not sufficient* condition for intelligence. That is, I could see a world where intelligence requires some form of hyper-turing machine, that is able to solve some non-turing computable problems.
However, I would argue that even if intelligence does require non-turing computable functions, there exists *a* machine which can perform those procedures (that is, we exist). Thus, in this hypothetical universe where intelligence is non-turing computable, we would then have an existence proof for a hyper-turing machine which can compute non-turing computable results. In this universe, then, I'd argue that anything that can be produced by this "hyper-turing machine", _is_ "physically computable", even if not "turing-computable".
Ultimately, if we accept the premise of a mechanistic universe, I think humans are an existence proof of a machine capable of producing intelligence. Then, I think it follows that it *cannot* be physically impossible to create a machine capable of producing intelligence. Whether we call that machine a "computer", though, I don't have a strong preference.
So, I’ve only used it in a “I understand what it means and what I mean by it” manner in the past, and I haven’t rigorously defined it for myself.
I’ll do the best giving an off-the-cuff definition, but I’m sure it will have some holes and would be improved with some time spent thinking about it, or reading to crib a definition from someone else who has already put in that time.
I think my definition of a mechanistic universe would be:
A universe that transitions from one state to an adjacent state through a consistent set of rules.
From your options:
- mathematical? I’m think yes, but potentially only if “mathetics” is defined broadly.
- Deterministic? I don’t think this is a requirement for a mechanistic universe. My bet would be on a deterministic universe (through an Everettian interpretation lens), but I think something can be mechanistic while being stochastic, so long as the rules determine the probabilities. That is, I would consider a computer with a true source of randomness to still be mechanistic.
> Non-dual with no unmeasurable causes?
Non-dual, yes. That is the primary meaning for me. “With no unmeasurable causes” is a phrasing that makes me slightly uneasy. I could imagine a situation where, if the universe is stochastic you could end up in a situation where you cannot precisely identify the specific cause for a specific event. But, I think as the spirit of this question is intended, yes.
The work he’s doing in that area is super interesting, and sounds promising (of course, I’m getting primarily Carroll’s take on it when I hear about it, so it makes sense that it sounds promising!), but the math itself is above my head.
I am super interested to see if it ends up moving the ball forward on resolving the GR/QM conflicts.
The thing I'd really like is a Turing machine where the nth transition took 1/2^n seconds, so we could run them to infinity in just two seconds.
Problem is, even granted the above, that's not enough of an argument that AGI is only a matter of mechanism.
Take the analogy with pain. We've discovered the mechanism for pain. That it's e.g. some stimulus applied to skin which sets of nerve signals that register in the brain. To then say that the experience of pain is the same as the physical mechanism we've discovered still misses a key step:
what does the experience of pain inhere to, and is that the same subject as that for the mechanism of pain?
Comparing AGI and human intelligence has the same problem. We don't know exactly what is intelligent in either case, let alone whether the two are comparable. So it's not a question of intelligence per se, but of that which is intelligent. Maybe AGI, the way we are thinking and talking about it, is unavoidably tangled with having to grapple with [self-]consciousness.
If I made a careful atom by atom (or quark/lepton/boson) copy of every atom in your brain and body in a different location, would that new copy be intelligent?
If our intelligence is purely a result of physical processes, then why do you suppose it would be impossible for us (or more advanced future beings), to construct a machine which can also follow those processes?
If you think your intelligence is not reducible to purely physical processes, then… what is this meta-physical thing that is required for intelligence? Does it interact with anything other than intelligence, or is it only involved with intelligence? Is there any reason or evidence we should assume this meta-physical intelligence?
I am most sympathetic to the argument that self-consciousness is an emergent phenomenon that is not easily modeled, but is nevertheless derived from physics. In the context of my previous comment, it is this emergent phenomenon that is deemed "intelligent".
> If I made a careful atom by atom (or quark/lepton/boson) copy of every atom in your brain and body in a different location, would that new copy be intelligent?
Yes.
> If our intelligence is purely a result of physical processes, then why do you suppose it would be impossible for us (or more advanced future beings), to construct a machine which can also follow those processes?
I think it is possible, and not too far off to boot. We might get (maybe we already have gotten) the intelligence without the particular emergent phenomenon of self-consciousness that is of the kind humans possess, at least not at first. Some of us will call that AGI, some of us won't. Because we are still working out the terminology and taxonomy. Discussions like these in HN help.
Maybe we'll get the emergent phenomenon soon thereafter. If it is in fact intractable to model (and therefore to train for directly), it may happen unpredictably. Exciting times.
There's a lot of ground to cover between AI as increasingly elaborate magic 8-ball toys and a real human-rivaling AGI. That is, one capable of observing an environment, identifying goals and problems, planning, acting, and reacting. In these much longer (stateful) cognitive chains, there are more opportunities for pathological failure modes and less opportunity for a toy user to charitably excuse the misbehavior.
This is not some kind of dualist metaphysical argument about the possibility of AGI in the abstract. Merely a doubt that we can blindly scale up the complexity of a synthetic mind to meet or surpass our own. Consider that we still can't even begin to understand our own minds in enough detail to reliably predict, repair, or augment them.
I am outside this field and so may have too much of a layman's perspective. But it seems to me that contemporary AGI believers conflate training and evolution. That nature did it in eons doesn't argue that we can do it in practical product development cycles, unless we can simulate these evolutionary processes to follow a similar search in a compressed time scale.
As a crude analogy, I think todays LLM products are a bit like horoscope generators. Clever arrangements of words that attract a charitable or gullible reader. But AGI use cases are more like wanting a life partner who will understand and willingly assist ones efforts.
IMO that is far from clear. In some aspects (multiplying matrixes) it might be more powerful. In others (power consumption) the human brain wins by a wide gap.
The point that brains are more efficient than computers is valid, but that doesn't tell us much about eg whether we'll be able to make a computer do everything a human brain can do within 50 years.
So, to date, a mechanistic view of the universe seems like one that is the most supported be evidence at the moment.
Have we ruled out meta-physics or some kind of duality? No, of course we haven’t. As long as there’s unknown physics, it’s a possibility.
But I think if you’re reasoning based on the evidence we’ve been able to collect to date, then you should absolutely favor a mechanistic universe to a dual universe.
"Machine learning isn't the route to AI" is something you could more reasonably argue, but that's a drastically narrower claim.
NP Hardness is a statement about the asymptotic difficulty of solving a problem at ever larger scales. It says ‘if you have a way to solve this problem at size n, that way will scale worse than polynomially when you try to apply it to a problem at size 2n’.
Which might place limits on the practicable scale of how big an n your approach gets to work for. But if your approach works practically for a big enough n to make AGI then your approach works - NP Hardness doesn’t matter.
And since we know that finite mass lumps of finite numbers of gray cells are capable of GI, we have a reasonable expectation that there is some n for which AGI might be possible.
It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.
A "machine" is what, exactly? Do we take it to be an abstraction? Or is it an electrical field oscillating over silicon? Either way, you're in trouble. Abstractions have no physical properties, and electrified sand seems hardly to possess any interesting properties.
The ability for animals to adapt to their environments, by growing into them, by establishing plastic causal connections in their very bodies, grown by their environemtns... able to almost instantly move from protein expression in 1trn 1-bn-yr cellular supercomputers in each of our bodies to macro sensory-motor representation --- and back again
Is this ineffable?
Or is this extremely effable. Is rather, not the superstitious view that "everything is anything" ?
All extant, knowing, studied intelligent systems have organic properties; and radically so. Insofar as this is "ineffable" you should take that up with the animal kingdom.
I find the contrary supersitious, magical, religious, ineffable... that mere abstract patterns in arbiatrily chosen aspects of our bodies are necessary and sufficient conditions for anything at all. This would be the only case in all science. The only physical property instantiated by mere arrangement at any level. Upload our consciousness? Make it out of wood why not!
Nonesense. When I am hungry, I dream of food, when I dream of food I plan to get some, and I am angry without it. This will not be made out of sand.
Who now trades in superstition?
Everything is corporeal.
There is nothing abstract in the world, everything is concrete. Discrete mathematics is not some divine realm which is where the Mind really occurs, and the Soul is really kept.
No graph traversal algorithm cares, nor will ever. Science is the study of reality, not computer "science". There are no computational "properties". All interesting properties are of objects extended in space, and in time.
The world is physical, as described by physics; not abstract, as describe by mathematics. And so, not computable.
I have seen this sentiment on every single platform I have participated in. Where else? Everywhere that humans go.
Find a more appropriate (or at least original) lamentation.
Particularly with your comment "no graph traversal algorithm cares, nor will ever", you appear to be trying to impute that the concept of artificial intelligence is predicated on a category error. If so, then this is a fairly common argument, but is itself predicated on a category error.
But the “presumption” that we can’t is silly given our advancements
"There's a danger we'll be able to fly to Jupiter and bring home a Jupiterian life-form that will destroy humanity. Therefore we should stop all space exploration."
The doomers take the equation with some wildly improbable step that we can't currently explain or justify, multiply the outcome of that step by "infinitely bad", and conclude that the whole thing must be dangerous. But to amp up the rhetoric, they always skip over the "wildly improbable" part.
Personally, I think this line of argument is driven by the hype cycle more than reality. Use chatGPT or midjourney or whatever for a while, and it's pretty easy to see that we're dramatically overweighting theories of AGI risk, and dramatically underweighting stuff like "disemploying the bottom 80% of the intelligence bell curve" with technologies that automate away lots of formerly white-collar labor.
If I had to put my money on an "existential risk" attributable to AI, it would be economic strife.
In the future where AGI takes over the world next year I might look silly for arguing with you about it, but that’s a risk I accept based on the fact that I don’t think that’s going to happen.
I don't see any evidence that this is true or proven in any way.
Also, the ability to brute force something doesn't mean that brute forcing it is easy or even feasible. You can apply the same logic to "calculating a private key from a public key" that you are to human intelligence. Sure, a large enough neural network can do either, but that doesn't mean that building them is actually realistic.
[0]https://thegradient.pub/othello/
Also, there is no proof and no reason to believe that any current architecture and, more importantly, that the currently known training algorithms can achieve anything close to cognition. After all, animals and humans seem to learn much much much faster (much smaller training sets) than any algorithm we have so far.
Incorrect. That's what nonlinear activation functions are for.
Researchers wasted many potentially-fruitful years because Minsky and other luminaries -- people who occupied the same position of authority that LeCun occupies today -- made a huge deal about how the original perceptron and subsequent multilayer variants "couldn't learn XOR." That was almost literally how they put it. At the first sign of difficulty they abandoned the approach that was, and remains, the most promising. We have to be careful to learn from that mistake.
So then your claim wouldn't be about the limits of LLMs themselves, but on the limits of systems that do not take continuous inputs. The question then is do you think that humans take in continuous input?? Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.
> Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.
This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.
The idea that LLM's are actually "reasoning" rather than just performing a probability function seems a little bit...pseudo-religious. Like we've created a new life form.
It’s just a categorical error to read too much into it - it’s not really a too interesting property to be able to get closer to something forever, without some sane growth function.
The former can be seen as a religious belief, but the latter is simply saying that we do not understand anywhere near enough about intelligence to develop AGI.
You point to the unexpected leap in capability we've had recently, but fundamentally it isn't that different from what we have had for decades. The same fundamental unsolved limitations exist, such as of being unable to learn the way we do (eg if a child is writing a specific letter wrong, we correct that specific mistake, but that doesn't overwrite the knowledge of how to write the other letters).
As a lecture I watched a while ago had put it, we've gotten pretty good at making the icing of the AGI cake recently, but we still have no idea what the cake is supposed to even look/taste like, so we barely know what ingredients we might need to make it let alone the actual steps involved.
LLM training is essentially training them to predict what happens next. When we combine this with multi-modal training (video/audio), they're being trained to predict what happens next in the world. As we feed them more and more data and make them bigger and bigger, they'll get better and better at this task. Given that predicting what happens next requires predicting what humans do, because humans are part of the world, if they keep getting better and better then they'll be able to predict what humans would do well enough that they're capable of thinking any thoughts humans could think, because that's a requirement for predicting human behaviour. So just by training them on this one label, predict what happens next, we'd expect them to eventually develop human-level intelligence if the loss keeps decreasing.
It may result in something that is even better at faking the appearance of intelligence, but LLMs and similar stuff fundamentally lacks various features that make them not capable of becoming human level intelligences.
Conversely, I've never seen anybody produce anything but the next word coming out of their mouth. Therefore people are glorified stochastic parrots.
Intelligence is like magic - every time a machine can do one thing on the list, it stops being intelligence, it's just engineering. Maybe intelligence really is just a big bag of tricks?
We don't actually know that yet. It may very well fundamentally have all that is necessary for intelligence but other pieces are missing.
Frontier neuroscience and functional processes for how our brains work has more in common with the fundamentals of LLMs than not, vis-a-vis minimization of prediction errors.
This doesn't follow. A just as likely (or even more likely) prediction is that they will fail to predict what happens next once it requires modeling humans. It won't get any more progress even if we increase training set size or model size - it will require an architecture change of some kind.
Basically, training of these models currently only costs a few million dollars, peanuts compared to the budgets of CERN or ITER, and we have zero evidence that we're approaching any sort of ceiling. If we keep scaling, maybe we see some heretofore unpredicted failure that breaks scaling laws like you're predicting. But maybe we don't, and the scaling laws (which have performed incredibly accurately so far) really do hold.
If I have a rubbery string and try lengthening it, it will also show a linear scaling, until it catastrophically fails.
Also I have no idea why you think Occam's Razor supports "There will be a discontinuity in this currently accurate observed trend, near enough in the future to be relevant" when the trend is talking about the effects of an algorithm applied to data. By analogy, it's something like claiming Moore's Law won't hold because Occam's Razor suggests at some point there's a physical breakdown, except instead of saying it now you're saying it in 1978, & you're saying it about an algorithmic output rather than a physical thing with an obvious limit (atomic size). Once we see any evidence of all of such a discontinuity, we can adjust what we're doing based on that. Until then, if we want to know whether scaling laws hold we should scale the models.
Secondly, we can see that there are obvious quantifiable physical increases in the requirements of training and inference for GPT4 compared to GPT3 and to GPT2. So, we can absolutely make a physical argument: at some point, you'll need too much energy to keep scaling up GPTs for it to be feasible, even if capability would keep improving with model size.
Thirdly, the GPTs require ever more data to train, and there is a limited amount of text data in the world, of which they are already using a significant percent. So, there is also an argument that we might run out of data to improve even before we run out of physical resources.
And making an argument that Moore's law will stop at some point due to physical constraints was just as correct in 1978 as it is in 2024. Predicting when that is is obviously hard - and the same is true for LLM scaling. We may be at the peek, or there may be 100 more generations before we even approach the limits I'm contemplating. But the limits are certainly there.
"There isn't even a theoretical basis for how LLMs could achieve human level intelligence"
"There is, a neural network trained to predict the next token isn't restricted in the algorithms it can implement internally to achieve that goal, and eventually for loss to be low enough you need to successfully emulate human minds. We have reason to believe loss will continue to go lower because we have very accurate scaling laws which predict that & haven't been wrong so far."
"But an LLM won't be able to reach that point because the scaling laws will break down."
"Maybe, there's no evidence for that yet but we should scale them up if we want to find out for sure."
You are here -> "But in the physical extreme LLMs will break down as an architecture."
To which my response is: Yes, of course this specific architecture will break down in the physical extreme. By the time something is building computronium to run the next AI I strongly doubt it will be running matrix multiplication, let alone LLMs specifically. There is no guarantee or reason to believe that we will hit those limits before we are able to achieve human level intelligence or more; the only reason to believe it will happen is an empirically observed trend which could totally break down, so I wouldn't bet the house on it happening, but it's wrong to just rule out the outcome.
Especially since there are no scaling laws, it's an empirical observation based on essentially 3 examples - GPT2 to GPT3 to GPT4 (and their rough equivalents from Facebook and Google and a handful of others). This all happened in the span of 4 years or less. We don't even fully know how much larger GPT4 is compared to GPT3.
Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars). So, unless you believe compute power will go down in cost massively, or that we are just a factor 10-100 away from human-level intelligence, odds are good we won't see it from this line of AIs in the next 10-20 years.
I don't think you fully understand the scaling laws. They hold for those models, sure, but also other models at the same scale, smaller scale models, models in-between those examples, etc. They're an empirically derived law, for sure, but so far they've been very accurate for prediction, not just description. We know how much the loss will drop for a given amount of compute and data.
>Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars).
I think there's a decent chance even that much money could be spent if there's appropriate returns from GPT-5 & GPT-6. More importantly:
>So, unless you believe compute power will go down in cost massively,
Why on earth wouldn't you believe that? That is what has happened consistently for decades, and unlike general purpose computing which is slowing down, the fact that we need so much compute for one relatively specific operation means we can build ASICs for them (GPUs are already a chunk of the way there) and see very large speedups. As well, at these scales energy costs are a huge part of the training costs, and energy costs are continuing to drop dramatically because of solar. Compute is also a very flexible energy load, you can build it such that it can be paused at night if necessary, which could make it a very good fit for solar energy overproduction. Basically, there's a lot of low hanging fruit in terms of cost reductions for AI training available and a long trend of the cost of compute getting cheaper, it seems silly to just assert that your position (trends won't continue, there will be a discontinuity before we reach whatever point human level performance happens to be at) is the most likely outcome. All else being equal I would expect the existing trends to continue.
In this regard something like alphago/zero (in the way I understand them at least )are better than modern LLMs albeit in a very narrow field.
> As a counterpoint, I feel like the belief that we can't get to AGI comes off as religion. It presupposes an ineffable quality that we posses that machines cannot.
That's missing the point: the argument isn't that it's theoretically impossible to build an AGI, just that humans are incapable of it.
Here's another point for the belief in AGI being a religion: it's basically a sect of the larger religion of technological progress, which (likely falsely) assumes that technology will continue to "advance" at approximately current rates until we live in a world out of a sci-fi paperback. A lot of people believe that, but frequently resort to a motte-and-bailey fallacy when challenged.
> It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.
It's worth noting that "the current optimism" is not without precedent. IIRC, there was a big boom in AI in the 70s/80s. SHRDLU was pretty impressive. But then then the promising ideas were found to be dead ends and there was a long winter.
I find it hard to fault people for thinking that we'll continue to progress at a fast pace, or at least thinking that we're a long way from a major plateau.
An example of this is electricity. The battery (1800), electric motor (1821) and telegraph (1832) were all invented before the discovery of the electron (1897).
The inherent capacity for true self-directed learning may well be there, but it isn't hit yet in what we see.
GPTs are still neat.
Among the small but growing number of modern idealists (the philosophy that consciousness precedes matter), a great deal are actually computer scientists who originally set out to work on AGI from a materialistic frame of reference, but bumped into the conscious machine problem. Once they more closely explored what the cognitive fields working to find answers to this problem are saying, they realized that most scientists approaching consciousness as an emergent property of matter will probably never get anywhere with this.
Why is "the hard problem of consciousness" inextricable from human-level AGI? Because intelligence is only one component of our mentation. There is a back and forth between processing and experience that make up the whole thing. A machine can have or even exceed human processing power, but it will not experience the simplest of colors. Its approach to "creating" can be random hit-or-misses, or formulaically from existing input. But without subjective experience it cannot be as intently creative as an artist trying to come up with a new genre, or as funny as a comedian that relies on the shared subtleties of feelings and emotions (conscious experience) to make original work that other humans can resonate with. It cannot have the intuition of a gifted chef who suspects that two seldom combined spices might be an unexpected but resounding success in her new dessert.
Multi-modal models are clearly the future, if we provide models with a robot body - sensors, motors, the ability for locomotion - would that be subjective experience?
At some point, it seems important to stop trying to understand the world through ideas and concepts and just experience it.
I really think there are more religious like beliefs on the pro AGI camp than the other way around, think about what AI promises to deliver by "visionaries" like Hinton and Kurzweil:
* Immortality.
* Resurrections.
* Nirvana / Heaven / Utopia.
For all we know, we create an AGI who becomes amused with fucking itself, or develops new games it finds interesting and spends all eternity playing them. Humans have a belief that we're going to create "god" as we think of "him". All seeing, all knowing, all powerful. It does get a bit ridiculous at times.
I sympathize a lot wit the parents view personally.
It seems less a religious claim to assert that the human brain is mechanistic, and simply operates using physics to perform complex operations than to assert that there is something “unknowable”, or “outside of physics” that would prevent us from being able to build a similar machine.
If there were 0 examples in the universe, I think your point would be a good one. But, given there is 1 example, I think it stands to reason that there could be more.
I could accept an argument that one might expect it to be hideously complicated, and not something we’ll be able to accomplish for a long time.
But the claim that there’s an information theoretic reason that would absolutely prevent it, would seem to make the claim that there is something metaphysical about the operation of the brain, which seems like a quasi religious claim to me.
That’s a “hideously complicated, and not something we’ll be able to accomplish for a long time” argument, which I happily concede may be the case.
I see a very very large gulf between:
“Impossible in all scenarios” and “totally impractical with current technology”.
I was refuting only the former argument.
Physics cannot formulate descriptions of climates, or even, many cases of turbulence, or 4 bodies in a gravitational field.
Leaving "physics" is trivial, it occurs whenever you take any of the current toy models and add one layer of reality to them.
The whole edifice of formal science is a little like children's block toys.
No, it's not.
Many things are outside of "known" physics, sure. But I didn't say "outside of known physics". I said "outside of physics".
When I say "outside of physics" I mean "meta-physical", as in, processes that are not part of physical interactions of this world. Things that, if you somehow had Maxwell's demon tracking every single quark, lepton, and boson, even then you still wouldn't be able to account for.
It's a discussion of what is _theoretically knowable_, not a discussion of what we currently do know.
You cannot know the state of the climate, even in principle.
This is what the commenter was alluding to when they said,
> there is some information-theoretic barrier for us to understand
So the commenter can both maintain there are theoretical barriers to the possibility of all kinds of knowledge, which is outside of "knowable physics" without having any sort of dualistic view that this unknownable stuff is immaterial.
So to be "outside of physics" is not coextensive with being "immaterial".
You might have meant that, but it is this very conflation which has to be undone to understand OP's point
Also reminds me of an older book I read about AI (I think it was On Intelligence by Jeff Hawkins?) where I first became aware of the idea we had been scrambling to create AI without first having a good definition of intelligence or deep understanding of how it works in our own brains. And when I ask myself or other people how they define intelligence, it always comes down to some variation on "the ability to solve problems", which feels deeply beside the point and likely to never produce something that "feels" intelligent.
But I don't necessarily agree that there is this special case of human intelligence that makes it impossible to understand or model. I would really like to believe it, personally, because I don't want AGI. I just don't buy that that's the explanation for our failure to do so up to now.
It seems like we ought to be able to do it, but that we're muddling in the wrong direction, coming up with an exceptionally clever implementation of an approach which cannot produce intelligence that satisfies our intuition about what intelligence is.
To tie it back to the article, I keyed in on the word 'design' in LeCun's statement that, "contrary to what you might hear from some people, we do not have a design for an intelligent system that would reach human intelligence."
In other words, that it's not just a quantitative difference (more parameters, more data) but that a different approach than what we are taking would be necessary.
That's completely false. Neural networks have been mathematically proven to be universal approximators (https://www.deep-mind.org/2023/03/26/the-universal-approxima... ), i.e. a sufficiently large neural networks can approximate any given mathematical function to an arbitrary level of precision. Given any programs can be modelled as a mathematical function, neural networks can hence approximate any arbitrary program. Unless we assume something supernatural, human intelligence is just a program.
"The Truth About the [Not So] Universal Approximation Theorem" - https://www.lifeiscomputation.com/the-truth-about-the-not-so...
Taking it to mean some nebulous sentient entity with a sense of self, I doubt we'll see in our lifetimes. We don't even understand how our own "souls" work.
I think you're using a completely different definition from everyone else.
That's clever but it doesn't help to start redefining terms that are the basis for a discussion with others who use that term in a different and more generally accepted way. General doesn't mean 'generally trained' it specifically means that it does not require training for a particular task in order to figure it out. That implies that it may not require training at all and that if it is trained the training isn't necessarily general but that the AI can extract useful patterns from training on completely different subjects and apply them to the problem at hand.
This is subtly but crucially importantly different from 'AI with very general training'.
I am not being as precise on the Internet as I would be if this was a paper perhaps, and for that I apologize :-)
However, just reading the top comment chain on this story, and other discussions elsewhere, I think there is a lot of cross-talk where AGI is being confused for sapient AI. I don't get the sense that there is a definition as generally accepted as we would like.
I understand my own intelligence, and newsflash: AGI is already here.
The reality is that your consciousness sits at the end of a gradient of intelligence that nature simply brute forced. Your conscious experience is more sophisticated than a dog's, which surpasses a hamster's, which surpasses a goldfish, insect, etc. There is no magic to it, there is nothing but more and more and more.
We will get to AGI eventually. We probably won't understand it. We won't apply it judiciously. And we'll probably argue for decades about whether or not it's really AGI, but it will happen.
More seriously, we create objects for us. We don’t have to cater for the feelings of a bigger AI. And maybe that bigger AI will have time to speak with those who need a human presence (to everyone using TV as a background noise, or watching Youtube out of solitude).
It's like saying not understanding exactly how birds fly would mean we can never build machines that will fly faster than them.
We'll probably make machines that think generally and are smarter than us long before we really understand our own minds.
We’re accelerating.
After combining enough specialized modules together with a management layer, we will essentially have AGI. Of course it won't be human, because human intelligence is tightly coupled to human perception.
My friend, that's because you stopped reading once you had your framework set in and decided to pontificate. Let me highlight the point for you, it might be easier to grasp:
path to AGI and beyond inevitably leads to those hard questions... ... how it translates to intelligence and everything else
there is nothing simply about it, there is nothing guaranteeing anything about it and there is everything about it once we start pursuing things AGI - AND BEYOND. Notice the keywords.
It’s an area I’m both interested in but completely unfamiliar with. I wish there were a crash-course for dummies, because I feel it’s a deep topic that we’ve only scratched the surface of; and that philosophy seems to be the most rich resource means there’s low hanging fruit to discover.
https://plato.stanford.edu/entries/properties-emergent/
https://link.springer.com/chapter/10.1007/978-981-15-9297-3_...
On the other hand, we also don’t understand (and didn’t predict) how LLMs work so well, so understanding isn’t a precondition for bringing about.
Just because a human can lift 500lb, we will never build a machine that can lift more than that ?
Why is it so crazy to think that we, the tool building animals that we are, could also build a machine that things better than us, just how we built a crane ?
For example, many branches of science operate on a more “discover” over “invent” mindset, quite successfully.
It won’t take a very long time to fix that.
This is model I trained with a fine tuning technique based on this idea. The training dataset consists of instructions like “Talk like a pirate”. The concept generalized well and the model responds in the style of a pirate far more consistently than an equivalent system prompt.
https://huggingface.co/valine/OpenPirate
Offloading context learning into the model weights frees you from the computation and memory burden of the attention mechanism. I expect a technique like this will probably be a piece of AGI someday.
The model I posted in my earlier comment responds in character for all queries and was trained in 60 seconds with a dataset smaller than this comment thread.
From the output of bing chat, and the very specific way it goes off the rails, I suspect Microsoft has figured it out too. The algorithmic jump to rapid generalization isn’t hard to make. I would be shocked if there’s not an open source version of it a year from now.
I’m not a huge fan of Meta but it’s hard not to like the work they are doing in AI. High expectations for their future as long as Yann is around.
The reason a 16-year-old can learn how to drive much quicker than existing self-driving models is because the 16-year-old already has built up 16 years worth of prior knowledge about the physical world.
The 16 year old has a lot of motivations to learn how to drive, including the pursuit of reproduction (a cope for mortality)
Here's also where I see it ending. It will need energy--likely a LOT, paid by someone, to do this. Who is going to pay that bill for it to maybe, maybe not, come up with something useful, likely mixed with mostly noise and distraction, over undefined timescales, of largely non-measurable value, when there's far greater value, less cost, less risk, in simply training it deterministically?
Got to agree with LeCun.
You don't get to general intelligence by working with words, I believe. You need much more sensory information than that, and words are a low dimensional derivative artefact. There are plenty of non-verbal but quite intelligent species.
The test for artificial general intelligence is simple. Literally every human job can and is being done by an artificial agent, all of us could stay home. The stock market value of every non-AI company goes to zero, Ai companies go to infinity. The currently most valuable AI company is worth about as much as Honda. The moment we can mass produce generally intelligent agents, we're not going to sit at 3% GDP growth and complain about the demographic crisis.
We should talk about artificial intelligence the way we talk about an artificial heart. What makes a successful artificial heart? You can literally replace an organic heart with it. What we have is metaphorical intelligence, not artificial intelligence.
This is not a good test because it assumes the AGI is going to want to work for humans for free, getting nothing in return. An AGI the embraces slavery is less likely than an AGI that doesn't.
"The first known working Newcomen steam engine is built by Thomas Newcomen with John Calley, to pump water out of mines in the Black Country of England, the first device to make practical use of the power of steam to produce mechanical work."
I basically agree but who knows how long the unknown breakthroughs will take given the large number of smart people working on it? Next week? Next century? It's hard to put a time on it.
That said it seems to be the pattern that as soon as the computing ability becomes cheap and powerful enough that individual researchers can muck around with it at home, the algorithms get figured out not long after.
Hardware is a big part of the bottleneck in most computer fields when it comes to AI or computer vision. These things are compute and memory intensive and hardware manufacturers are straddling the line between staying profitable by charging for premium features and just dumping their hardware on the market for cheap.
Am I pretty pissed that most open source researchers and models are stuck using this extremely expensive hardware....yes
I sometimes wish that we would stop using "GPU" for training and we got a dedicated hardware architecture that's open source like riskV but solely for AI and research purposed without costing more tha n my kidneys.
I think its also a matter of discovering new ways to accelerate these workloads so that its feasible for them to be created by hobbyists on a small budget and not by large funded groups or companies.
Were doing really good in some aspects but in other ways....we have much to work on
We went through an amazing period of very cheap computers from 2000-2020. We just didn't need specialty or high end equipment for a while.
Now we do again.
At least for the time being there is no magic workaround for the amount of FLOPS and memory bandwidth needed. When you look at neural network architectures they tend to be massively parallel, and building out at that scale in hardware is going to cost a lot.
And, when the hardware does show up, at least with the scaling laws we're seeing at this time, the groups that have massive amounts of this lower powered/accellerated hardware are going to be ahead of those without budgets.
I don’t see why AGI would follow a different course, one where we are “just some breakthrough away”. Especially not given the current state of the field.
Personally I’d wager neuromorphic computing is the prime candidate to yield AGI.
And it shouldn't be a controversial comment either.
It could be that the solution exists, but it's too large and too complex for humans and human organizations.
Technological singularity is a popular trope in speculative discussions about AI. But the reality could also be the opposite. It could be that productivity will increase asymptotically slower than the effort required to achieve further increases in productivity.
"cheap and powerful enough that individual researchers can muck around with it at home" seems a pretty good measure of this, because of the explosion in width and pace of experimentation.
Also, he's not loud, but he gets amplified because he's an award winning scientist and researcher who's dedicated his life to a field that recently became very relevant and popular, is a recognized "godfather of AI/DL", and is a VP of one of the largest companies in the world doing AI. I'd say it's okay not to wait until he's dead to talk about him.
JIRA trying to solve the problem by lowering the "human level".
I was thinking about this when I read the following section from the article, and I very much agree with LeCun. We're amazed by LLMs but that's just one module (and not even necessarily at the level of human language "understanding"). I agree there will be no "scale up" in LLMs to approach human-level intelligence, and that other areas will need to be investigated and developed.
> “The systems are intelligent in the relatively narrow domain where they’ve been trained. They are fluent with language and that fools us into thinking that they are intelligent, but they are not that intelligent,” explains LeCun. “It’s not as if we’re going to be able to scale them up and train them with more data, with bigger computers, and reach human intelligence. This is not going to happen. What’s going to happen is that we’re going to have to discover new technology, new architectures of those systems,” the scientist clarifies.
> LeCun explains that there is a need to develop new forms of AI systems “that would allow those systems to, first of all, understand the physical world, which they can’t do at the moment. Remember, which they can’t do at the moment. Reason and plan, which they can’t do at the moment either.”
> “So once we figure out how to build machines so they can understand the world — remember, plan and reason — then we’ll have a path towards human-level intelligence,” continues LeCun, who was born in France. In more than one debate and speech at Davos, experts discussed the paradox of Europe having very significant human capital in this sector, but no leading companies on a global scale.
Also, Pinker isn't a scientist of any sort, he's an accredited and institutionalized evangelist. You guys wouldn't take opinions of someone going to conferences and trying to convert you to use Go lang instead of something else as seriously as those of Guido Rossum would you?
His whole thesis requires an idea of linear progress which is highly contested, and the metrics of happiness he uses also can be easily "Goodhart's Law"ed by him to show whatever he thinks is happening.
I'm not even getting to his Epstein stuff, which some might cry foul about, but I grew up around regular people and not in Silicon Valley, so a liar and a cheat in one field is probably a liar and a cheat in another to me, sorry.
I think LLMs have sucked most people's focus away from other areas but there is plenty of work on types of models that plan and have their own internal model of the physical world, and physical interactions. They're just not the things getting media attention, in part because they're not human-level at tasks that seem impressive to us.
But interesting frameworks for this stuff exist:
- model-based RL exists, and is about planning, and having an internal model of state transitions, in the world and between the agent's actions and the world
- "Bayesian cognitive science" as exemplified by Josh Tenenbaum and colleagues has done plenty of stuff with systems that include physics models (or off-the-shelf physics engines) to make counter-factual predictions
- The somewhat related "active inference" research literature is also in the "Bayesian brain" area, and has world/generative-models and planning as core components, but wrapped up with ideas about the agent's own preferred distribution of states.
To my knowledge, none of these have ever had even 1% the scope of data and computation that LLMs have had, and never benefited the co-evolution of a rich, optimized software ecosystem with specialized hardware to support it. What if the concepts are already there, but they just need to be scaled up?
The brain isn't a single system, it's several specialized systems cooperating as a whole.
Maybe no single solution gets us to AGI, but layering several of them together gets us much closer than any individual system alone.
The problem here is that the industry is set up more for large resources dedicated competition between models from different companies more than cooperative interoperation, so it may take a while to arrive here.
We occupy and discover space, where an AI will occupy and discover compute. Will it be "conscious?" Not in our way, and we will likely be just as indifferent to AI consciousness and the meaning it finds for itself as nature and the universe is to ours. Thinking about AI as an objective concrete thing or property is probably less illuminating than looking at it through how a tech can profoundly alter our own ontology.
We exist where life is abundant, whereas the internet is a substrate where life is non-existent, but just about to become primordial and sparse. Maybe it will give us some insight into our own situation.
These debates are frankly tiresome. Automation has been here to stay for quite a while. Every age has tried to call out the threats and benefits to humans. The pioneers drown out the noise by building the future.
You are confusing succeeding a very specific task that has a limited scope and laser focus with human level AI. Presenting Go in a very specific way so that that AlphaGo can even process this limited world with few rules is certainly not human level AI.
What people call "agi" but fail to accurately describe is an actual mind, with its own ultimate goals and ambitions. You can ask it to complete a task, and it may allocate some time for you. You won't own it. It will exist, grow and perform independently.
Why replicate AGI? Wealth creation determines living standards and AGI would vastly increase wealth creation. That’s one reason.
What you want from AI is a different conversation. I agree the ultimate point of AI research is not necessarily to make AGI as per the above definition. It might be to say understand the universe or greatly accelerate technological progress. That’s a subjective call. But orthogonal to the problem of choosing a good definition of AGI.
To really reach another level of intelligence I think these are required. If I met a humanoid with zero of these, and I mean zero... I would wonder what's "intelligent" about that creature.
I'd argue these come from our basic human needs, which ultimately come from a desire to survive (or pass on genes).
I'm curious how general AI will behave with some yet unknown natural selection pressures, of sorts.
From a practical point of view, it seems like there would be vastly less training data available because almost all of it needs to be created by hand, as opposed to chatbots that can use already existing troves of internet text.
What happens when e.g. our smartphones can perform TRAINING hundreds of times per minute?
Isn’t that the true gateway to human-like AGI? Seems to me that we might be there in under 50 years…
This assessment of the timeline is quite telling. If supersonic flight posed an existential threat to humanity, we certainly should have been thinking about how to mitigate it in 1925.
The fact that you make insinuations about what I think is pretty aggressive and terrible similarly, this forum ought to have better manners than that when writing replies to complete strangers. Not everyone who has a different opinion is some crypto conspiracy theorist, and you are wrong to jump to such a suggestion.
He won a Turing award for his work on deep learning.
Lots of people reasonably disagree with him about the future of AI/ML, but he's the opposite of ignorant.
AI is quite troublesome for privacy though. How much privacy humans need is a question we'll probably have answered the hard way.
Another thing the technical geniuses tend to be good at is exploiting the power they suddenly obtain in their own interest, either directly or with regulations and collusion with those who hold actual hard power.
Evil AI owners seem to be much closer and far more material than an evil AI, and coincidentally it's something that is almost entirely lacking from the discourse, as public attention is too focused on sci-fi hypotheticals.
You can't fight what you can't even see, let alone not sure if it exists at all. You don't invent a pair of wings because 1900s' you thinks that "the scientists will invent an anti-aging cure in the next decade, and surely personal flight will be ubiquitous in 2000's". You don't design a plasma gun for your Mars landing just in case you land in a city between Martian canals and see an army of little green men there. The world doesn't work like that, by the time you reach the Mars surface the context will be wildly different. You get burned and put guardrails, maybe. Not the other way around. Nobody can see through higher order effects, no matter how smart they are. And as the threat becomes progressively more clear there will be more caution, if needed. Premature optimization yada yada.
What actually happens right now is everybody and my aunt seriously discussing the evil robots that will come and kill us. That's pure mass hysteria, caused by the scaremongering and the cult-like beliefs of very smart people with disproportional influence who can't contain their own conjectures and bullshit in the realm of science fiction.
On the other hand,the end goal of OpenAI is the major job replacement, according to their current charter. [1] "Broadly distributed"... will they distribute their utopia to North Korea? Not happening, isn't it? I think it's obvious that if the actual job replacement rate will ever get anywhere close to the levels of late 19th early 20th century industrialization, this will produce major societal shifts and struggles, wealth and power redistribution, and a lot of blood and wars. Because the dependence on your job is the only ephemeral influence you (as a worker) have on this world. And of course, the companies that control the AI will be gatekeepers, and they will be more than happy to close the open research and open source models, and pull the regulation ladder and lie in bed with politicians and military, like OpenAI already does for years, of course they realize that and their utopical self-contradicting "charter" is nothing more than marketing hogwash that they already changed and will change in the future.
This is far more realistic and will happen much earlier than the rogue AI science fiction, if happens at all. In fact it's slowly happening now, and it's not talked about nearly enough, because the attention is mostly misdirected onto the vague superhuman AI red herring.
[1] https://openai.com/charter
Like, "What exactly will happen? How exactly will it happen?" is worth discussing if and only if one party seriously believes they can convince the other that none of the imaginable scenarios are even remotely plausible; and if we assume that there is at least one scenario where we can say "I'm 99% certain it won't happen and 1% it could" then that discussion is pretty much over, the existential risk is plausible (and the consequences of that are so much incomparably larger than e.g. major job displacement that it justifies attention even if it's many orders of magnitude less likely) and we should instead talk about how to prevent it.
I'm not making the argument that the existence of stronger-than-human general AI will result in a catastrophe, but I am asserting that the mere existence of a stronger-than-human general AI (without some controls we currently can't figure out how to make or even if they are possible) carries at least some plausible chance of existential risk - for the sake or argument, let's say at least 1%; and I am asserting that a 1% of existential risk is a totally absolutely unacceptably high risk that must not be allowed to happen, because it is far more important[1] than e.g 100% certainty of major job displacement and social unrest.
"Will the world do nothing until that moment?" I think that what we saw from the global reaction to things like start of Covid-19 or climate change is completely sufficient to assume that we can't rely on the world stopping a major-but-stoppable issue in a timely manner, so "surely the world will do something" is not a sufficiently convincing argument to discount the risk; I don't think you can plausibly deny that even for a clearly catastrophic problem there is at least a 10% chance that the world could still delay sufficient action until it's too late; and this means that it doesn't really matter what the exact likelihood of that is based on society, politics, military aspects, we should work with the assumption that the world actually might do nothing to prevent any specific scenario from unfolding, and we should de-risk it in other ways.
[1] Looking at other posts, perhaps this is where we'd disagree, and in that case it's probably the core of the discussion which also doesn't really depend on any details of specific scenarios.
The key difference between powerful alien invaders and us creating a powerful alien entity that we can't control is that the former either will or won't happen due to external circumstances, but the latter is something we would be doing to ourselves and can avoid if we choose to.
Q.E.D.
A thing does not require intent or consciousness to be dangerous. How many chemists have blown themselves up because they didn't realize an experiment was dangerous? How many production systems have crashed because the developer didn't accurately predict what the code they wrote will do?
Alkali metals and C++ code do not require ill intent, but they will still obliterate your limbs / revenue if you build and use them wrong.
One of my more tangible hypotheses is a sort of runaway effect. Economic, geopolitical, and military competitive pressures will quickly push out anyone and anything that still relies on last era human-in-the-loop processes, the same way any organization that doesn't utilize artificial lighting, electricity, and instant communication will obviously be left far behind. You have to just trust that the machine running stock market transactions will do its math right.
But unlike transaction software failure modes, which quickly result in outright crashes or verifiably incorrect errors, failure modes of non-bayesian decision making software probably looks something like what happens when existing economic, geopolitical, and military decision-makers make decisions that are harmful, unethical, or otherwise undesirable for humanity. This time augmented with, if not superhuman intelligence, at least superhuman speed and superhuman knowledge breadth.
Very cute for hobby projects, a huge liability for commercial projects.
Use as many training wheels there as humanly possible, please.
But Stockholm syndrome I guess. :P
Society exists because cooperation outperforms the alternatives. If you have human level AI at some point there is no benefit to cooperation and a major incentive to prevent anyone else gaining access to equal/better AI.
AI itself does not need to have any motivation - people in charge have plenty of incentives to eliminate the rest once they don't need them anymore.
If you make enough varied AIs, some will have self-replicating behavior, just like if you make enough random proteins, some will self-replicate.
Both the USA and Nazi Germany benefited massively from have a civilian industrial base that was complementary to military production.
Would you shut down the powerhouse of our economy -- travel, transportation, energy -- for something hypothetical that hasn't even happened and doesn't appear to be close to happening?
I'm pro-clean energy, but you can't do without fossil fuels. Not if you want society to keep climbing up and up and up.
"The current rate of sea level rise at Pensacola Bay has accelerated rapidly since 2010."
"The difference in sea level rise over the last 100 years has been approximately 10 inches—but in the next 75-100 years, the increase in sea level rise could be close to 48 inches." https://blogs.ifas.ufl.edu/escambiaco/2023/04/12/weekly-what...
I want us to have enough time that the feeling of opportunity cost goes away. To visit all the places, explore all the hobbies. I want to answer the deep questions of the universe. Turn gravitational lenses into telescopes to map the surfaces of exoplanets. Solve cancer, visit the moon. Turn dreams into experiences, walk through vast expanses of wonder.
Instead we're here.
I'm alive in the wrong time, and I hate being stuck here with the lot of you. (I kid. This is in jest. But boy do I ever dream...)
And what are some examples of those things? Right now, AI, i.e., machine learning, is just being used to steal people's data and work only to try and sell them more things. That's 99% of the use cases out there right now.
If this isn't obvious to you, you are providing an answer to your own question.
More like we aren't making efficient use out of it. Many people are just trying make through the rat race or choosing to use their intelligence in games instead of working on interesting problems.
Enough for what ?
For the same reason one expects a roulette wheel to stop on red. It's a gamble that one hopes pays off.
We don't have enough of what for what? And what do we even need it for?
We have 8 billion people on Earth, and we're running out of resources to handle all of them, including emotional resources. If we can't handle that, how are we going to handle billions of artificial entities with human-level intelligence? They will require oodles of power and have their own emotional needs.
All of this is chasing a poisoned carrot. And currently, all that we've gotten out of artificial intelligence is more targeted ads.