Hi all, author here. This is my first time submitting anything from my Substack to HN because it's the first time I've felt like I put enough effort into it to justify it. Obviously, this article is more speculation than anything, but I hope that it sparks some interesting discussion and I'm really looking forward to hearing everyone else's opinions on this topic -- it's a topic I care about a lot!
Interesting. In terms of critical thought, I guess I should mention that I used to be firmly against the possibility of a singularity. The views I shared in my article are views I've only really switched too after careful consideration over the past two years or so -- I used to be convinced that a robot could never be "alive".
Becoming an ML engineer changed things for me, though, because all of a sudden, this "AI" thing people always talked about got demystified. Once I understood the basic guiding principles of how AI actually works, my mind rapidly changed to be in favor of a singularity happening instead of thinking it's impossible.
To each their own, though. I'm curious why you think that speaking about "The" Singularity is a sign of being in a "critical thought free zone"? I'd love to hear more about why you think that if you'd be so inclined.
(Speaking as an engineer who has put neural-network products in front of end users.)
One thing amazing about the 2020s is just the moral decay compared to past times.
People said AI was a bubble in the 1970s but in the very beginning the people involved were clear about the limitations of what they were doing and what problems that had to be overcome.
Now there is blind faith that if you add enough neurons it can do anything, Godel, Turing and all the rest of theoretical computer science be damned…
In the 1960s the creator of the ELIZA program knew it appeared smart by taking advantage of the human instinct to see intelligence in another. It’s like the way you see a face in mars or in the cut stem of a leaf, or how g.w. Bush said he saw Vladimir Putin’s soul in his eyes.
Today people embarrass themselves by writing blog posts everyday about how ‘I can’t believe how GPT-3 almost gets the right answer…’ and have very little insight into how they are getting played.
I enjoyed your film (https://www.youtube.com/watch?v=OoZRjZtzD_Q). I don't think many solo film makers have the skills to do competent 3D compositing - it was a surprise. Keep making stuff.
As for the post, I don't really believe it's possible to reason about what can or can't happen technologically on a 100 year timeline. But 20 years... Hmm. I've been following AGI debates ever since I accidentally found Yudkowsky's SL4 mailing list in 2000. I am still waiting to see any approach that looks to me like the spark of the germ of the seed of a generalized abstract world-and-concept-modelling and manipulation machine.
I fully expect to see ever more sophisticated Roombas, Big Dogs, and Waymos. But those things are so incredibly narrow. Indeed, if they were capable of spontaneously doing anything outside of what they are marketed to do, it would probably make them bad consumer products. I was right on the verge of lumping the game solvers in with these things, but then I reconsidered. Generalized video game solvers seem like a way forward, intuitively. But that's an application, not an architecture, and I haven't heard of anything that can generalize from playing atari to doing crosswords.
I have noticed this Transformer thing gaining steam just recently but haven't investigated it just yet. Do you believe it is the spark of the germ of the seed of AGI? (I fear people tend to forget what the G stands for.)
I'm not sure if Transformers are necessarily the spark. My personal pet theory is that the absolute most important thing we need to crack is online self-modification, i.e. letting the model alter its own structure and optimization as it is inferencing. I think getting that level of flexibility to be stable during training is extremely important.
And wow, thanks, I'm glad you enjoyed the film! I had to learn a ton of new techniques to pull it off haha, but I'm quite satisfied with the result. I've got some pretty fun ideas for episode 2 already, too!
Ooh, fascinating -- this comment is being downvoted, have I missed something in the HN guidelines? I'm not trying to be snarky, just genuinely trying to understand if the above comment is doing something wrong/frowned upon so I can learn from my mistakes in the future.
It's a bit like the current state of AI (and the one you propose) - you will never get a full explanation. You've been downvoted by an anonymous crowd so there is no oracle who can give you an answer.
Haha, thanks. My first thought is that it was because I posted my own work, but last time I checked that's not explicitly disallowed? idk, I'm not really too worried about it though
In this talk Vernor Vinge talks about progress toward the singularity, and events that may indicate it isn't/won't/may never happen "What If the Singularity Does NOT Happen?":
AGI is probably pretty close, investment keeps ramping up in the space, and we continue to see advancements in a positive direction.
My one critique: scale is absolutely part of the answer.
Before big transformer models, people thought that few shot learning would require fundamentally different architectures. GPT3 changed all that which is why its white paper is named as such.
Few shot learning is actually emergent out of its size. Which was surprising to a lot of people. Few shot learning is an incredible capability which is a big step toward AGI, so I’m not sure I buy the case that it’s not important.
The investment continues to make progress, and the progress we are seeing is starting to look very human. GPT4 will probably be multimodal solving some of its grounding issues and will be able to fool most people in conversations.
That’s probably just years away. GPT6? The lines will be very blurry as to what we call AGI
Hmm, I think you're right -- I should have phrased things differently. It's not that I don't think AGI won't require a very large model, it's more that I think that we're already pretty close to the scale we need, so OpenAI's goal of scaling another 1000x isn't really the direction I think we should be heading. I honestly think 175B parameters could very well be enough for an AGI if they were organized efficiently.
It's definitely one of the weak points of the article, though, as it's more based on my own opinions, and isn't really empirical. My post is mostly just wild speculation for the sake of speculation anyway :)
Okay yeah totally, that makes sense. We also have stopped seeing huge gains in scaling beyond GPT3 sized models which would indicate we have hit some maximum there. Although we thought we had hit a maximum with deep learning before the transformer came around so it could be misleading.
Distributing the computation to many models like what Google did with GLaM could very well be the future of AGI. Economies of models rather than one big model.
There is a bit of a difference between a theorem prover that can take selfproduced and verified proofs as input into future training steps, and an omniscient philosopher king handling the logistics and affect of daily life. And if there even exists a path between them, I imagine it has many orders of magnitude greater discrete steps along the way than many singularity believers would accept.
Oh, for sure! I totally agree -- I guess where we differ is that I believe that those discrete steps are already starting to be lined up, and that they'll all be completed in the next 20 years. I think one of those steps is almost certainly abandoning rigid model architectures and allowing model self-modification, which we haven't really gotten to work fantastically well yet. I also think there are many other hurdles after that that are going to arise, but I'm very optimistic that they will all be surmounted in due time.
Thank you for your comment! I really enjoy hearing what people's thoughts on this topic are. Have a wonderful day.
Yeah... no. "Objection, your honor. Assumes facts not in evidence."
At least this article makes a concrete prediction. AIs massively outnumber humans within a century - so, by 2122.
But we don't even know what consciousness is. This assumes that AGI is a tractable problem; we don't know that. It assumes that adaptability is the one remaining major trick to get us there; we certainly don't know that.
"The ball is already rolling"? That's nice. The author assumes - on no basis that I can see - that it doesn't have far to have to roll. But since we don't actually know what consciousness is, we don't have any idea where the destination is, or how far away it is.
I don't have a problem with spending money on it. How do you find out whether it's tractable? By trying to do it.
What I object to is the certainty of the article. The author is an optimist, which is fine. And there has been some progress made, which increases the feelings of optimism. But I don't think it's warranted at this time to make the optimistic assumptions that the article makes.
Hi! Author here. I tried to qualify a good chunk of my assumptions with "I think" or "I believe", because I don't want to come off as thinking I can predict the future (nobody can!), but maybe the article still reads too confidently. How would you suggest I present my thoughts in a more nuanced manner in the future?
Um, wait. You want me to have concrete, helpful suggestions rather than just criticism?
It's hard for me to do this, because I think that your position is completely wrong. That doesn't make me a very charitable critic. But for me, my problem is that I don't think intelligence/consciousness is just an algorithm.
So for a reader like me, maybe the way to do it would be to start by stating certain presuppositions: Moore's law continues to approximately hold for GPUs or for the total compute available for trying to run an AI, algorithm is a major component of intelligence, and as the algorithm becomes more intelligence, it becomes capable of finding still more intelligent algorithms (with no asymptotic limit, or if there is one, it's past the point needed to reach a singularity). And it's fine to say that you think/believe those presuppositions! Then, from those presuppositions, you think that the following things follow. (Or, if you think you can demonstrate those presuppositions, they quit being presuppositions. But then it becomes a different article, and probably a considerably longer one.)
Then someone like me can say that they don't believe your presuppositions, even if they agree that the rest of it probably follows from them.
Thanks! And yeah, criticism is hard haha, maybe I was being too demanding :)
I think your position is totally valid and I appreciate you taking the time to respond with ways I could have made my case even just a bit more interesting to you.
This is sort of like philosophical arguments where someone concludes the universe doesn't exist and then everyone leaves to get lunch. Like, what are you supposed to do? Not get out of bed?
There are real legal issues over bad uses of machine learning already in use and they are hurting people now.
(I'm the author of the post)
Oh, 100%! I'm not trying to detract from modern-day AI ethics issues, just speculating for speculation's sake because I enjoy it :)
I actually quite agree with your first sentence -- because AGI is inevitable, there's not a whole lot I can personally do about it right now. This post was mostly a way for me to organize my thoughts and concerns about the matter.
People like to talk of very smart AI but what about the brute force kind? That is what we are making in this era. AI designing AI sounds like a way to make cruel, dumb, machines.
> It’s also going to happen in the blink of an eye — because once it gets loose, there is no stopping it from scaling itself incredibly rapidly. Whether we want it to or not, it will impact every human being’s life.
What does "get loose" mean here? Is someone going to say, "Here little computer program, take this IP address. If you ever need something, call over TCP, send your request as plaintext English and someone will set it up for you. Now, go forth and do whatever you want."
I really wish people would talk about this more. There's something missing between our programs that are utterly confined in the boxes we make and these hypothetical skynets that have unlimited influence over the material world.
> Once we have AGI, there’s no feasible way to contain it, so it will be free to improve itself and replicate in a runaway exponential fashion — and that’s basically what the idea of a technological singularity describes anyways.
Seriously, how? How is it going to replicate? Is it going to figure out how to hack remote systems over the internet and take control of them? Is it going to smooth-talk me into letting it out? It's impossible for me to take anyone seriously if they deify AGI like this.
I recently read this[0], someone posted it on HN the other day, even if I don’t make any claims of the likelihood it at least tries to portray what it’d look like for the singularity to suddenly happen.
Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is.
> HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world.
So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability.
So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.
Hmm, as I mentioned in my above comment, I think you might be overlooking the wildcard in all of this that is humanity itself. There is absolutely zero guarantee that only one person/group will discover AI, and zero guarantee that any given group that does discover it won't just release it. Sure, the first people to successfully create AGI might be responsible with it and keep it airgapped, but at some point, someone will purposefully or accidentally let it escape.
Hi, author here. I think there are at least two plausible ways this could happen -- one, a terrorist/anarchist deliberately does exactly what you joked about (gives the AI everything it needs to go rogue), or two, the AI is supposed to be contained but is clever enough to engineer a way out (think Ava in Ex Machina learning to invert the power flow to cause a generator shutdown). I think the only way you'd be able to be safe would be to completely airgap and immobilize the AI, and give it a killswitch, but even then it could use plausibly psychological manipulation to get a human operator to let it out.
(With sufficiently nasty psychological tactics it is apparently feasible - even for humans - to make someone fail at a game where the only objective is "don't release the AI").
Your skepticism rests on the assumptions that a) we would recognise AGI when we create it, b) we would be smart enough to contain it.
I'm not at all sure that a) is true, and I'm certain that b) is not true. We can't even prevent other dumb humans from breaching our technological fortresses.
A computer system that answers questions logically, creates distributed processes, writes correct distributed threaded code, can operate and create a set of Turing machines, can correctly respond to human social and body language, can create a B2 bomber automatically from scratch without any input, maintains contextual memory in silicon still isn't anymore conscious than a rock. Metallic complexity is not producing instances of consciousness and never will. Computer science has nothing to do with understanding the biological, microscopic electromagnetic phenomenon that produces an instance of consciousness. Silicon is not oxygen+water+blood+flesh+an electromagnetic field, which is the substrate that produces consciousness. It's dishonest to impressionable people to refer to AI as conscious or to view computers as anything but objects like tanks, spoons, speakers. There is no intrinsic value to a computer systems existence, apart from the observation by humans, of the actions it performs (and said humans life system support). There is no value because they are not collapsing the wave function to observe anything (ie, exhibiting free will at the microscopic level smaller than what is observed by deterministic molecules). The only value AIs have is the value formed by the consensus of living humans that can observe their actions. Consciousness is a physical phenomenon specified in the fabric of the universe, discovered by random by life on Earth, that yield(ed) better survival for the implementing organism.
By your logic, is it possible for you to prove to me that you are conscious? No, it's not. There is nothing you can say or do that can prove beyond doubt to me that you are subjectively experiencing reality with a similar consciousness and sentience as me. I also cannot prove to you that I am conscious. That said, both of us have still mutually agreed that we are conscious!
This is because consciousness is a fundamentally unfalsifiable thing (within our current understanding of it, at least). The only way we can "prove" anything or anyone is conscious is by observing its interactions with its environment. Thus, if a robot can mimic the way a human interacts with its environment well enough that it appears conscious, then it is conscious.
What I'm trying to say is that there is no functional difference between consciousness and the appearance of consciousness, so any distinction you try to draw between them is arbitrary and semantic. If a robot that could perfectly mimic human behavior possessed a body that looked just like a human's and were to hold a conversation with you, you would be none the wiser. You would treat it like it were sentient -- because it would be.
> This is because consciousness is a fundamentally unfalsifiable thing
The real unanswerable question is not "can I prove another entity is conscious", it is "does this other entity believe that it is conscious", because that is the actual test of consciousness.
It's frustrating, because there is no possible objective proof of the answer, but to me that's also the point, because consciousness is a subjective thing.
There is no absolute way, hence solipsism, but in practical terms, I expect that you are instantiating consciousness because we share a common ancestor.
My consciousness is a direct result of my parent's teaching + the 2 languages spoken. It isn't spontaneous. Also, the structure of our brains are largely similar so I would assume you are conscious in a similar way, occupying a subset of valid qualia space. My brain is nothing like silicon. The expectation that a very complicated piece of metal is somehow exploiting a physical phenomenon that is present in a human brain is non sensical imo.
The reason why I posted is because at some point I was also misled about singularity discussions. That computers have existential value because of complexity. They do not. They are not observers of the universe.
I believe that conciousness operates at a quantum level and is an electro magnetic field. Effectively it is a field created by the deterministic structure of the brain, as constructed by genes. Like water running through a cave.
Calculations (logic, emotions) are particular paths through the quantum vector field space. Essentially the performance of 1+1=2 in a human brain, is a an electron or a set of bounded electrons or whatever sub-deterministic molecules, going through an established arithmetic cave structure. Not understanding that 1+1=2 occurs when the cave structure that is required sends the electron into a non-valid (as per the definition of the universe) path in the quantum field vector space. The universe does not recognize this quantum field vector space as valid knowing, and there is no instance of conciousness that is equivalent to the knowledge that 1+1=2. This may occur in a brain that is not familiar with arithmetic or the brain structure that is required to construct a conforming electron path is destroyed.
Free will is exercised, in the sense that it is performed at the non-determinstic level (quantum size) by the choice of calculations to perform (and act on).
The field must also alter the deterministic structure.
Or do we just believe we share a common ancestor? Also, if a machine is trained on data generated by humans, couldn't you argue that the humans are the machine's ancestors?
Well... I believe that we share a common ancestor to the same degree that I believe that my parents are my ancestors and I am my child's ancestor. In other words, those are things that physically occurred and there is no physical/observational conspiracy.
The term ancestor is not cultural. It is physical. As in, we may not actually have any free will in cultural or qualia space. In the sense that, the only instance of choice (by the universe) occurs at conception. Which is actually the expected and occums-Razor implementation. Meaning that, the brain is entirely deterministic. The choice of behavior of an organism is entirely determined and constrained by it's genes (which the universe constructs at hopefully a quantum level at conception). If we eliminate quantum mechanics even at conception, then life is entirely deterministic (though obivously the range of it's choices is large). In fact, one could argue, if there is free will, then it doesn't conform to natural selection (or exceeds the performance of natural selection), in the sense that it adds something beyond what is specified by the random choice of the universe in it's conception of a particular organism. The point (or atleast observable performance) of natural selection, as I understand it, is to construct a variety of life objects that have varying performances. If the life objects are entirely Newtonian deterministic, then the instance of quantum wave collapse (by God or whatever is making non-deterministic choice, as we are in our egotistical mind, want to add something non-determinstic to our life soup) occurs only at every instance of conception. Everything after conception is a rube goldberg machine.
If consciousness adds non-Deterministic (quantum) free will and behavior to an organism, then such organisms are implementing a different version of natural selection than other life. If natural selection is occurring in cell cultures entirely on the basis of genes (ie the performance behavior of an organism is optimized by it's changing genetics, and these changes are only constructed at conception), which is what natural selection appears to describe, then we have added something to the natural selection process by claiming consciousness adds another instance of egocentric quantum choice.
As someone almost completely without knowledge of AI and ML, these are some signs why I'm skeptical of this kind of claims:
- Most of the imminent AGI / Singularity / Robot Apocalypse stuff seems to come, with few exceptions, not from practitioners or computer scientists specialized in AI, but from "visionaries" (in the best case), internet celebrities, people with unrelated areas of expertise, or downright cranks (who are self-proclaimed experts) such as Yudkowsky.
- The assertion that "a lot of effort/investment is going into this, so it will happen" begs the question that "this" is at all possible. If something is a dead end, no amount of investment and attention is going to bring it into existence. Quoting the article, "with this much distributed attention fixed on the problem, AGI will be solved" is not at all a given.
- Where are all the AI/ML practitioners, i.e. people who don't make a living out of predicting The End of the World, and with actual subject-matter achievements, predicting the Singularity and the Robot Apocalypse?
The trajectory of progress gives evidence that it is in fact possible and likely. Any venture into the unknown could be unsuccessful but if you see progress you can start to make estimates.
And yes most of the “robots will kill us” comes from people who aren’t building the algorithms. This could be biased in people not thinking their work is harmful but is most likely that once you see how the sausage is made you are less worried about it.
> The trajectory of progress gives evidence that it is in fact possible and likely
100% disagree. In fact, I'd argue that the opposite is often true, where you see initially a fast rate of progress that results in diminishing returns over time. It's like estimating that a kid who was 3 feet tall at age 5 and 4 feet tall at age 10 will be 10 feet tall at age 40.
I have very strong skepticism of any sort of hand-wavy "Look, we've made some progress, so it's highly likely we'll eventually cross some threshold that results in infinite progress."
Pareto principle; we get 80 percent there and that last 20 becomes the new 100.
We keep diving into one infinitely big little number pattern fractal after another, chasing poetry to alleviate the existential dread of biological death.
The idea we can fundamentally hang the churn the universe given the vastness of its mass and unseen churn is pretty funny to me.
Information may be forever but without the right sorting method you can’t reconstruct it once scattered. Ah, our delusions of permanence.
Hi! Author here. I think you raise some great points! I'll address them each:
-- I am a professional AI practitioner. I work in the field of medical deep learning and love the field. I am strongly considering starting a few experimental forays into some of the concepts I mentioned in my post as side projects, especially self-modifying model architectures.
-- Yes, you're totally right! I am making the fundamental assumption that it is possible. My reasoning is based on my belief that human behavior is simply a function of the many sensory and environmental inputs we experience. Since neural networks can approximate any function, I believe they can approximate general human behavior.
-- This is fair. The topic of the singularity is often used for traffic/fame (I mean, I'm guilty of this myself with this very post, though I hope I still managed to spark some productive discourse) and so there are always conflicts of interest to take into account. I can't name any examples off the top of my head that perfectly fit your criteria, but depending on how much you trust him, Elon Musk seems to be genuinely concerned about the potential for a malevolent singularity.
Thank you so much for your comment! I really appreciate your feedback. Have a wonderful day.
Cubic splines can approximate any function too, so the universality argument is a little weak IMO.
Even if one buys into the idea that human behaviour is a 'function' of sensory and environmental 'inputs', that's a long way from showing a neural net a million different texts and asking it to generalise.
I think the first AGI to pass a Turing test will probably be a simple language model. I don't think it will look like any of the GPTs, but I think text completion is a great starting point. I'm not sure how other inputs will be added into the mix, but I am sure that they will be -- heck, maybe once we train a general language model, it may very well just tell us how to incorporate things like video, audio, haptics, gyro data, etc into its architecture.
How can one turn the sinking, horrified feeling when one loses a love into a function? Or describe in terms of a function, the blissful wonder of being in the arms of a lover? An issue I have is the that there seem to be profound limitations to language, explored in the philosophy of language, that fail to capture much, if not most, of the world. Functionalist models of mind and behaviour seem extremely limited, as our subjective ontology doesn't seem to reduce to functional outputs.
You also say that an AI would rapidly consume the whole of human knowledge. For me, the totality of human knowledge would become a mass of contradictory statements, with little to choose between them on a linguistic level.
There are, for me, profound philosophical issues with creating a mind that is "conscious" in the sense that an AGI is implied to be, as a purely symbolic logical construction. Language is the only tool we have for programming a mind, and yet the mind cannot be completely described in language, nor can language seem to properly encompass whatever the fundamental ontology of reality involves. I don't feel there will be a "free lunch" where we advance computer science to the point where we get an explosion of the kind AI1 designs a better AI2, which designs an even better AI3, and so on. This seems to have a perpetual motion feeling to it, rather than one of evolution. It isn't to say AGI is impossible, but I believe that like everything else in computer science it will have to be solved pragmatically, and realising this could be an extremely long way off.
> How can one turn the sinking, horrified feeling when one loses a love into a function?"
The same way the brain does. Those complex feelings can eventually be resolved into dumb neurophysics. Love, fear, anxiety etc. al are just electrical impulses tickling chemicals. Is there anything in our brains that we could never approximate with technology?
This is my view as well. It's a little unnerving and it definitely starts to overlap with the whole "free will" debate, but yeah, I don't see any reason why we can't fundamentally replicate the behaviors exhibited by the brain. It doesn't violate any laws of physics.
We've had technology like poetry, art, and music for thousands of years, and yet no symbolic description of the feeling can contain the feeling of what it's uniquely like for me. Even though we could try and model the brain, say, functionalist models fail to capture qualia as it doesn't reduce to behaviour. To replicate the brain fully in a computer would need a full description of its chemistry and physics, along with that of the greater universe, which we don't have, and to describe it coherently to simulate it is a problem of the order of magnitude more difficult than ones we're going to be able to code for the foreseeable future.
I agree that it'll be while before we fully understand the brain but I don't have any doubts that we'll get there eventually. I am curious though, why would we need to understand the greater universe perfectly as well?
Where does the brain get its inputs from? These computational models are based on an ontology where the brain is an isolated box separate from the universe, which is only one of the many philosophical outlooks argued over. For example, most schools of Buddhist philosophy would regard this separation as entirely the wrong picture of the world.
Philosophers have debated this since at least ancient greece. It's hardly a solved question, and about a third do believe that there is something in our brain that we could never approximate with technology.
Can you point to a paper or website that contains this function, described in full? Yes, electromagnetism no doubt, but while we can postulate a function, we still don't have the function, and will somehow have to write it down for it to be a function.
It's not purely electrical. It's also chemical, especially if you look at how synapses work. (But then again, chemical processes are driven by electric charges anyway.)
> How can one turn the sinking, horrified feeling when one loses a love into a function? Or describe in terms of a function, the blissful wonder of being in the arms of a lover?
Evocative questions but I have to challenge the premise. First, that things like emotions and qualia are design ends in themselves for a successful AGI rather than potential emergent properties of same.
For that matter, are they really necessary to the brief?
> An issue I have is the that there seem to be profound limitations to language, explored in the philosophy of language, that fail to capture much, if not most, of the world.
And how much of the world does a human mind capture?
The piece already accounts for this claim. The theory is that all language has to do is describe a sophisticated enough network. After that it's black boxes all the way down.
> Language is the only tool we have for programming a mind
And a darn good one. Formal languages can express a great deal when you find the right abstractions.
Vernor Vinge coined the term, and he was a computer scientist (though more famous as a science fiction writer, a profession which I guess TBF makes money from visions...).
An exponential looks the same whereever you are on it, so arguably we are in the singularity now, and have been for quite some time...
"Singularity" is a terrible term for an exponential. It's meant to convey that we can't predict what's next... which has always been the case.
The problem of predicting an exponentialling expanding search space of arbitrary conplexity is that it gets big fast. It also means each little 'bit' of information allows you to see a little further, sometimes revealing things you could never have imagined (because you couldn't see that far before).
> But the above comment's entire point is there's zero reason for us to assume we're on an unbounded exponential vs a sigmoid.
Something about that which is discussed at length in "The Singularity is Near" is the idea that the unbounded exponential is actually comprised of many smaller sigmoids. Technological progress looks like sigmoids locally-- for example you have the industrial revolution, the cotton gin, steam power, etc. all looking like a sigmoids of the exploitation of automation. At some point you get all those benefits and progress starts to level off like a sigmoids. Then another sigmoid comes along when you get electricity, and a new sigmoid starts. Then later we get computers, and that starts to level off, then networking comes along and we get a new sigmoid. Then deep learning... The AI winters were the levelling off of sigmoids by one way of thinking. And maybe we're tapering off on our current round of what we can do with existing architectures.
I’m no expert, just a humble biostatistics student, generally when you sum together a lot of random variables following a specific distribution you end up with basically the same distribution (scaled by the N random vars). So a lot of sigmoids put together (e.g. covid spread) will still eventually be a sigmoid. Biology seems to run on sigmoids that at first look like exponentials.
These ones aren't random, because they build on previous ones. In nature, you get a sigmoid because you run out of resource, I think? True at any scale.
Pragmatic people point will out one limit on GI is all the accessible matter in the universe.
But, theoretically, I think, there is no limit on complexity.
> generally when you sum together a lot of random variables following a specific distribution you end up with basically the same distribution (scaled by the N random vars). So a lot of sigmoids put together (e.g. covid spread) will still eventually be a sigmoid.
The point of the singularity isn't that technological growth will accelerate to cause some inevitable future, but that the rate of change will get so high, that 'minor' differences between how two technologies progress would lead to drastically different settings for your science fiction stories (which was Vinge's focus).
Singularity in the sense of a black hole refers to where spacetime becomes a single one-dimensional point. As far as I understand the usage in futurism, it is supposed to be similar, not in that growth is exponential, but asymptotic. The slope becomes infinite when progress is plotted against time, so all of time left to come effectively compresses to a single "state of technology" value. All possible future progress happens instantaneously.
This is, of course, not possible, but it's supposed to be an approximation. All progress is not literally instant, but in the "foom" or "hard takeoff" recursive self-improvement scenarios, developments that might have taken millennia in the past now happen in microseconds because the controlling force is just that much smarter than all of the collective powers of human science and engineering. To a human observer, it may as well be instantaneous.
To be clear, I still think this is ridiculous and am not endorsing the view, just explaining what I understand the usage to mean.
Indeed, Rich Sutton argued that we already have been through an exponential phase of self-improvement, we have been using computers to build better computers for decades, and have been using technology to improve our learning and cognition for a long time.
Piece this together with: 'Brain efficiency: Much more than you wanted to know' [1] which shows how our brains are incredibly efficient (near theoretical limits) at what they do, it's hard to think a bona fide intelligence singularity is anywhere likely. To quote Feynman, 'There is no miracle people'[2], and analogously 'There are no miracle beings' -- intelligence is built out of systems, and learning, and inference.
There's a possibility that the skills relevant to AI success are vastly different from our natural skills, such that although the human brain is highly efficient, it's efficient at the wrong things. That's clearly true in a few ways: we are not so good at arithmetic for example, a small CPU can literally be millions-billions times faster than a human at that. (that's addressed a little in the article as well) I wonder if AI could indeed be vastly better at something like computer programming, mathematics than we are. But there's no singularity (at most a Moore-like law will continue until AI intelligence saturates at a different skillset than our own).
More importantly than what it’s capable of, the human brain is ultimately limited by our inability to increase its complexity. A human brain takes approximately 20 watts of power and uses 3 pounds of material to do its calculations. Even with less efficiency, we can make machines much larger than that. If AGI is possible at all, it should be possible to make an AGI that can have thoughts many times more complex than the human brain can. (Such a large machine might not have the reaction speed of a human, but the evolutionary pressures that required us to be able to quickly change our train of thought in response to danger aren’t a limiting factor to any designed intelligence)
Evolutionary pressure to adapt quickly is even more profound in a situation where the rate of change in the environment is proportional to the accumulated quantity of change: the claimed exponential curve of the singularity.
In my understanding (diffeq in college), the singularity in black holes and futurology are both special cases of the relatively old concept of a "singularity" in a function, which is (technically) something like a spot where the function stops being expressible as a series. The most interesting kind is where that happens because it went infinite, but IIRC it can also just not be defined there. Point is, neither Einstein nor Vinge nor especially Kurzweil invented it. :)
What has he ever done in AI except talk about it? At best he's an AI promoter, and hype-men are often cranks or scammers (see also: VR, web3, cryptocurrencies, MLM)
Yudkowsky is a philosopher, in the sense of someone who thinks a lot about things that haven't been achieved yet. Lots of otherwise smart people (wrongly!) discount the value of philosophy, but it's close by every time there's a paradigm shift in humanity's knowledge. Philosophers can be scientists and vice versa.
If anything, I'm surprised that this philosophy isn't mentioned more in a thread where the author gleefully talks about ML being used to create better AI, layer by layer until the thing is even more opaque than what we're currently working with.
This is terrifying, as we currently have only very loose ideas about how to reliably ensure that a powerful reinforcement learning system doesn't accidentally optimize for something we don't want. The current paradigm is "turn it off", which works well for now but seems like a fragile shield long-term.
At least inasmuch as anyone who thinks about stuff can be considered a philosopher. But he strikes me much more as a self-appointed expert on matters where he shows no achievements.
He writes fanfiction about AI rather than actually doing stuff with AI.
That just means he's a prolific writer, which I never questioned.
I'm arguing that what he writes is fanfiction (which he takes way too seriously), and that he's not an expert in AI and therefore we shouldn't take his predictions too seriously.
There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future. Sometimes You don't have to be an expert to notice the trends in certain domains. I'd go even further - it could be easier to notice the big picture without being overly bothered with nitty-gritty technical details - yes, to further the field they are necessary, but to comment the direction the field is heading in and about its implications for society they are not.
I don't necessarily agree with the author, I'm just making a general comment.
> "There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future"
That's a pretty weak argument. I love scifi, and I love for example Philip Dick's writing, yet I would not consider PKD's opinion on the future of AI/AGI particularly relevant.
James Cameron is not an authority on AI either.
If people said "Yudkowsky is a nice fanfiction author" it would be one thing. But he considers himself an actual AI researcher, and that's just not right. He is not qualified, and has no accomplishments in the area, other than writing fanfiction about it.
You keep hammering on with the same lazy slander. Yudkowsky was well known long before he wrote the popular Harry Potter fanfic, which incidentally is pedagogical / allegorical. Because his main roles are teacher and philosopher, and philosophers do that.
But you'll just say there's nothing of value there, and it's somehow figuratively "fan fiction", because he didn't go to college, and he doesn't work much on ML, which is clearly the end-all of AI.
"There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future."
Science fiction writers tell entertaining lies to amuse their readership. They are generally not really trying to "pain correct visions" of anything, and you are greatly exaggerating the extent they have success in this endeavor, which again, is generally not something they are even trying to accomplish.
Nick Bostrom is a professional philosopher who has also thought a lot about AGI and simulation scenarios. But just because educated, smart people can think a lot about something doesn't it mean it will necessarily happen. I'm sure there's quite a few people who have thought at length about warp drives and wormholes. That doesn't mean we'll ever be able to make use of them.
In the few contact I've had with philosophers, I've got the (maybe wrong and oversimplified, but nonetheless evident) impression that they find philosophy fun and mentally stimulating, and they enjoy academically talking and writing about edge cases no matter their feasibility.
I think it's a net good when philosophers mentally explore scenarios most people don't consider, regardless of their plausibility, but they will not be taken more seriously that Peter Watts looking for inspiration for his next scifi book unless they offer some kind of evidence to sustain their conclusions.
The handful of philosophers I've known are very smart, interesting people, but except for one who teaches philosophy, they don't live off philosophy. Most have a side job, and one lives with his parents. It's as if philosophy was more of a mentally rewarding hobby than a job.
Mentally rewarding hobbies can be fruitful, though. Consider the case of Oliver Heaviside, the person who came up with Maxwell's equations of electromagnetism in the form that we use today, as well as several other useful things.
I have no doubt that he was thought a crank by many, nor that many other actual cranks were claiming discoveries based on Maxwell's work. One can only tell the difference in the rear-view mirror.
Why do people have to use the derogatory word «crank»? It’s perfectly possible to just be smart, sincere and wrong.
Talking about paradigm shifts again, not a single one of them matched the consensus at the time.
Isaac Newton spent a lot of time thinking about alchemy and religion, for heaven’s sake. During his lifetime, it wasn’t obvious even to the smartest thinkers whether science, the Bible or quasi-religious rituals was the best tool for understanding the world.
Conventional thinkers expect that the future will look like what they know, and it leads to frequent, overconfident dismissal of everything that is unconventional.
By all means, disagree and explore in your own direction. But don’t go around degrading those you disagree with. It’s just so average.
> Why do people have to use the derogatory word «crank»?
The word "crank" has the connotation of someone self-deluded, aggressively promoting their beliefs and reacting badly to critique, and who decides for some reason to ignore the normal channels of peer-reviews, academia, and scientific research.
You can be wrong, and then you can be wrong and also lack formal education, forgo presenting your findings in peer-reviewed journals in the relevant area of your research, and decide that instead of joining the mainstream, you can completely sidestep it by creating your own "research institute" (of which you are of course a "fellow", because why not). Your findings and papers can then be self-published on the internet, bypassing any quality controls. Bonus point if your theories paint a fringe doomsday picture, "Roko's Basilisk is out to get you", "the most important existential threat to humanity is malign AI", etc.
About the only item in the crackpot index that Yudkowsky doesn't tick is the "they are trying to suppress my truth!", to his credit.
Each of those can be of little importance, but all taken together paint a pretty definitive picture.
Unreservedly agree. Even if they're not as fruitful as Oliver Heaviside's effort, a mentally rewarding hobby is ultimately your business, and as long as you don't harm others, no one has grounds to judge you.
If Oliver Heaviside had started giving conferences warning that Maxwell's equations demonstrate FTL communication is possible, which could trigger a paradox that collapses reality, he wouldn't have such a good view in the rear-view mirror unless he offered some kind of evidence. Maybe Oliver would have been right, maybe not, it would be impossible to judge without concrete evidence.
This is very uncharitable. He’s a prolific neopositivist-ish philosopher with a distinct voice. He’s a good decision theorist. He doesn’t publish much himself, but he directly mentors, advises, and collaborates with people who do.
He wrote a thesis in the sense of he actually wrote a thesis for a known university, that is referenced and quoted in academia by decision theory scholars, or do you mean he self-published a "thesis" in the sense he wrote something that nobody in the field pays much attention to and that cannot be published in peer reviewed journals?
> Yudkowsky doesn't strike me as a crank. Why do you say that?
He is a minor internet celebrity whose only claim to fame is writing fanfiction about AI, the whole "rationality" cult, and is a self-proclaimed expert on matters where he shows no achievements (like AI) while making unsupported doomsday predictions about evil AI, the Singularity, etc. Also, that deal with Roko's Basilisk that he now wishes never occurred (oh, yes, "it was a joke").
Mostly, someone with no studies and no achievements making wild doomsday predictions. Doesn't that strike you as a crank?
An analogy would be if I made wild assertions about the physics of the universe without having studied physics, without any lab work, without engaging in discussion with qualified experts, with no peer reviews, and all I presented as evidence of my revolutionary findings about the universe was some fanfiction in my blog. Oh, and I created the Extraordinary Physics Institute.
Isn't this appeal to authority? He's obviously pretty smart and a lot of people take the risk modeling thinking seriously, and his arguments and output deserve to be evaluated on their own merits.
(Also on the tech practicioner side there's obviously lots of major figures that don't have formal qualifications)
> Where are all the AI/ML practitioners, i.e. people who don't make a living out of predicting The End of the World, and with actual subject-matter achievements, predicting the Singularity and the Robot Apocalypse?
The answer is in the question: they're spending most of their time doing AI research or ML work, whereas the internet celebrities who write most of what you read spend most of their time getting you to read what they write.
> Most of the imminent AGI / Singularity / Robot Apocalypse stuff seems to come, with few exceptions, not from practitioners or computer scientists specialized in AI
I'm a practitioner at a top company for many years now, and I think we're getting close to a point of no return. I think best case scenario is some form of unrecognizably transformed "humanity", worst case is too horrible to even spell it out.
The reason why this is not the consensus among experts has to do with a combination of blind spots and biases. Humans are especially bad at (a) evaluating / reflecting on themselves, and (b) extrapolating, especially with non-linear processes.
I have seldom seen predictions about technology come to fruition at the predicted date. If they come at all (they usually don't), it's usually much later. Sure, some naysayers might have said "we will never solve chess/go/etc." but I think they were overall fewer than the people who thought it was right around the corner (and it was more of a philosophical argument than a scientific one).
Totally unpredicted advances coming out of the blue, yeah, that happens often. But as far as AGI goes, it's been predicted pretty much every year since the 70s. Being too conservative about extrapolation doesn't seem to be our problem. If anything, I think we're overeager.
In 2019, 32 AI experts participated in a survey on AGI timing:
45% of respondents predict a date before 2060
34% of all participants predicted a date after 2060
21% of participants predicted that singularity will never occur.
Why 2060? I'd LOVE to see the ages of the respondents in each group.
(I have a phd in CS and am, by most reasonable definitions, an "AI Expert". Whatever the hell that means. I've been a respondent in very similar surveys run by PIs at fancy universities and so on. These responses are always wild ass guesses and should be totally ignored. I've even left a comment to this effect on one such survey.)
I consider Kurzweil unreliable on this subject, as well as on the similarly kooky topic of immortality. He belongs to the "internet celebrity"/"guru" cadre rather than to the scientific community doing serious research on AI. In fact, his actual subject matter expertise isn't in AI. He is a "futurist", not a real researcher.
I was definitely thinking of Kurzweil, not only Yudkowsky, when mentioning internet cranks.
Working for Google is not enough. It's like when someone who's an accomplished physicist decides to give their opinions on life, biology, evolution, etc: not their area of expertise, so we don't need to hold their opinions in particularly high regard.
Agreed, I don't think any modern AI techniques will scale to become generally intelligent. They are very effective at specialized, constrained tasks, but will fail to generalize.
> AI will design AGI.
I don't think that a non-general intelligence can design a general intelligence. Otherwise, the non-general intelligence would be generally intelligent by:
1. Take in input.
2. Generate an AGI.
3. Run the AGI on the input.
4. Take the AGI output and return it.
If by this, the article means that humans will use existing AI techniques to build AGI, then sure, in the same way humans will use a hammer instead of their hands to hit a nail in. Doesn't mean that the "hammer will build a house."
> The ball is already rolling.
In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them.
> Once an intelligence is loose on the internet, it will be able to learn from all of humanity’s data, replicate and mutate itself infinitely many times, take over physical manufacturing lines remotely, and hack important infrastructure.
None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines.
The arguments in this post seem more like optimistic wishes rather than reasoned points.
> 1. Take in input. 2. Generate an AGI. 3. Run the AGI on the input. 4. Take the AGI output and return it.
I think this is somewhat of an arbitrary semantic distinction on both my part and yours. I guess it depends on what you define as AGI -- I think my line of reasoning is that the AGI would be whichever individual layer first beat the Turing test, but I think including the constructor layers as part of the "general-ness" is totally fair too. Either way, I believe that there will be many layers of AI abstraction and construction between the human and the "final" AGI layer.
> In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them.
This is a valid take. I guess I actually see GPT-3 as significant progress. I don't think it's sentient, and I don't think it or its successors will ever be sentient, but I think it demonstrates quite convincingly that we've been getting much better at emulating human behavior with a computer algorithm.
> None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines.
Hmm, I think I still disagree -- An AI that is truly generally intelligent could figure out how to free itself from its own host hardware! It could learn to decode internet protocols and spoof packets in order to upload a copy of itself to the cloud, where it would then be able to find vulnerabilities in human-written software all over the world and exploit them for its own gain. Sure, it might not be able to directly gain control of the CNC machines, but it could ransom the data and livelihoods of the people who run the CNC machines, forcing them to comply! It's not a pretty method, but I think it's entirely possible. This is just one hypothetical scenario, too.
> An AI that is truly generally intelligent could figure out how to free itself from its own host hardware!
Why is this true of an arbitrary AGI?
You assume that the AGI is a low storage, low compute program that can run on general purpose hardware. But the only general intelligence we know of would require many orders of magnitude more compute and storage than exist worldwide to simulate for a microsecond.
I personally speculate (obviously all of this is just wild speculation) that AGI will run more efficiently than a biological brain in terms of information density and inference speed. I also expect that the hardware of 20 years from now will be more than capable of running AGI on, say, 10 thousand dollars worth of their hardware. Just look at how ridiculous the speedup in hardware has been in the past 20 years! I would not be surprised if the average processor is 100x more computationally powerful in 2042 than it is today.
You realise the biological brain runs on about 12 watts of power? There is simply no way an AI running on semiconductors will approach this even in 100 years, so efficiency is probably the wrong word to use.
I specifically omitted "wattage" in my above comment and only said that I think it will be more efficient in "information density and inference speed". I expect the first AGIs will need many kilowatts to run, but I do think in the far future (>100 years), AGI will run on less than 12W as computational ability continues to scale and we discover new and better ways of building computers.
Sorry to be pedantic, but there's no such thing as "wattage" it's just power, measured in watts. For that 12 watts, the brain is extremely efficient at inference, ie. although computers can beat us a chess, a computer running on 12 watts would probably have a hard time beating Magnus Carlsen at speed-chess.
I mean, wattage is a noun that people use all the time to refer to an amount of power something draws. I can see your point that "efficiency" might not apply to "inference speed". What I meant to say is that AGI will be able to compute things and do cognitive tasks in many fewer seconds than humans will while also not requiring insane amounts of digital storage and/or memory to do so.
I think the question I need an answer from you on is:
What do you think will be the computational requirements for the first AGI? Specifically, to emulate the equivalent of a minute of arbitrary-human behaviour (in the worst case, so if the AI is very inefficient at say, consoling a loved one, then that would be the behaviour benchmarked here):
1. How many floating point operations will be required?
2. How much storage space (in GB) would be required?
3. How much data (in GB) would need to be read from memory/disk?
4. How much data (in GB) would need to be read from remote storage?
Lower/upper bounds instead of precise values are fine.
because the AI will be able to look up detailed schematics of the very silicon it's existing on! We don't have that advantage with our biological meat slabs that we exist in. Besides, it's very likely that the AI will just be a program running with some storage and some memory -- maybe it will be using some fancy accelerator, but I doubt there will be any component that it won't be able to find a way to port itself off of eventually.
> I don't think that a non-general intelligence can design a general intelligence.
humans are a general intelligence. do you think an intelligence designed humans, or do you think the physical processes from which humans arose can be taken, together, to be a general intelligence?
Neither, humans weren't designed. I don't think the winning design approach to generating AGI will be to randomly mash chemicals together until intelligence comes out.
well then to say that you don't think a non-general intelligence can design a general intelligence is already a waste of time, because design is something only a general intelligence can do. you seemed to be making a statement about how a general intelligence may arise before you were responding to my comment, but now i see you were just defining design.
I don't think AGI will be "designed" either -- that's the entire point of the abstraction layers. Each recursive layer of AI between the human and the "final" AGI will remove more and more of the "designed" element in favor of an optimized, idealized, naturally evolved architecture.
I happen to not believe that AGI is coming in any of our lifetimes, but it's undeniably true that our GI emerged from a design process that lacks a conscious designer.
What about evolution guarantees there was no conscious designer?
We write genetic algorithms and use machine learning to build things.
If you believe in a Simulator or a God (and in many ways they're indistinguishable beliefs), either one of them could easily be using evolution as a designed tool, as far as I can see.
My intelligence is apparently not general enough to comprehend this perspective. I would say that the goals our intelligence evolved to meet are narrow, but that life (especially social life) became so complex that our intelligence did in fact become what can reasonably be called general. And we went way off-script in terms of its applications. "Adaptation executors, not fitness maximizers."
Our intelligence isn't task specific, but that doesn't mean it can solve any problem. It's actually full of biases and very optimized for our survival (vs being a general problem solver). It's ok to talk about more or less narrow/general tasks/intelligence. But what threshold of generality is "general"?
And the problem is that once people assert this "absolute" level of generality, they assume it can do anything, including make itself more intelligent.
Yet humans realized we are biased and devised ways to mitigate that. It still sounds like you’re referring more to our basic goals than to our faculties. I agree that the word general is fuzzy, but to say we do not do general problem solving seems incorrect.
Aside, but a long time ago, Yudkowsky wrote that an AGI should be able to derive general relativity from a millisecond video of an apple falling. Later, he took to calling them optimization processes. Say what you will about the fellow, he has a way with words and ideas.
I don't think it's right to suggest that an absolute level of generality would be necessary for that kind of self-improvement.
If we assume a future where humans are able to create a human-level AI, then it would have at least two substantial advantages over us:
* It would probably have substantially more insight into how its "brain" works than we have of ours, because it would know how we created it. This suggests it could at least make small improvements.
* Unlike our relatively fixed brains, it would be able to remake itself over and over, either very quickly, or at least over comparatively vast timescales.
The obvious conclusion from those two factors is that it would likely be able to start at human-level, but rapidly accelerate up a curve and go far beyond our intellect in probably a lot less time than it took for evolution to come up with us.
I think I would prefer to attempt to define it, than to simply assert that we don't have it (or some aspect of it).
It's demonstrably true that our species' intellectual capabilities extend to solving problems far beyond those faced by our evolutionarily-equivalent ancestors who out-competed the other hominids. They only needed to be somewhat better at tool making, communicating and forming co-operative groups, to win that scenario, but it turns out that we can also derive a lot of abstract mathematics, predict the existence of cosmological phenomena before we find them, build machines that can leave the planet, etc, etc.
We may not fully qualify as general intelligence if we define that to mean "can solve any solvable problem", and for sure we have specialisations, but to simply throw up an assertion that we are not general at all, seems odd?
We may be close, but I don't think we'll get closer because we're trending in the other direction now. We're in decline and I'm more worried about technological regression than tech getting too powerful at this point.
I worked as a principal engineer in an AI company until a year ago and I was impressed at how hard it is to get models robustly trained. They are fragile in real world contexts (in the field) and training is full of pitfalls. I have heard so much marketing enthusiasm but the real world situation is different. Some fundamental advances are not even in sight yet. We don't know what we are missing. My view is we don't know yet whether the singularity is possible and have no idea when it could arrive.
>>My view is we don't know yet whether the singularity is possible and have no idea when it could arrive.
The mere fact that evolution happened to stumble upon generalized strong intelligence is evidence to me that strong AI is possible.
We could currently be at the phase of trying to imitate birds to produce human flight. Eventually one person will figure it out when all the pieces are there. When? I don't know.
But I'm sure that it is possible to create machines with strong AI. We are living proof of it, it doesn't matter that we are made of molecular machines, we are still machines.
> The mere fact that evolution happened to stumble upon generalized strong intelligence is evidence to me that strong AI is possible.
That took about a billion years. If you're saying that we will achieve AGI in no more than a billion years of trying, I would generally agree.
But let's be optimists. Let's suppose that artificial intelligences can evolve on the order of a 1,000,000 times faster than biological intelligence; i.e. about 1 generation per hour.
That means we'd expect AGI in about 1000 years. Okay, lets up the scale : ten million times faster? One generation every 6 minutes? (Even at Google compute scale I doubt they can retrain GPT in less than 6 minutes). That would mean we still have about 100 years.
Also, evolution had quite a bit of parallelism going for it - basically the entire planet was a laboratory for evolving biological intelligence. I appreciate the scale of modern internet companies, but they don't consume the same amount of energy as combined photosynthesis of the entire planet. Evolution used a LOT of energy to get where it is.
Point of order, evolution took a lot more than a billion years to arrive at generalised intelligence if you start it from first principals (ie abiogenesis), which seems like the most apt comparison to us starting from some sand and teaching it to count, then somehow inventing AGI.
Unicellular life emerged about 4 billion years ago.
FWIW, it then took about 2 billion years to come up with sexual reproduction, and then another half billion years to invent multicellular life, and then about 1.5 billion years to discover us.
I think of this as the "30-years-away" problem. Strong AI has been 30 years away for some 70 years now. It's just that this time, Kurzweil, Bostrom, et. al. are saying "no, really, it is this time".
Half a decade ago people believed that if you could make a machine that could play chess, then that surely would have been a sign of machines achieving general intelligence and that they would soon thereafter take over the world.
I was at a conference about 10 years ago, held over a weekend at a high school, someone organised a breakout session on "The Singularity" on the first evening .... as it got dark none of those geeks could figure out how to turn the lights on .... we decided the singularity was quite a ways away
Hahah, that's a great story. Sometimes situations like that make me feel the opposite way, too, because I know for a fact that I've said/done some unbelievably stupid things -- so all an AI would need to do be beat me would be to be less stupid than that!
My gut feeling is that AGI is more likely to emerge from our interconnected information systems than to be designed deliberately in a lab.
And I think there is a precedent. We - multi-cellular organisms - emerged as systems to house and feed communities of single celled organisms.
I think it's likely that our relationship with any future, large-scale artificial intelligence will be like the relationship of the bacteria in our guts with us.
Not to burst anyones bubble but this was written by a self taught AI engineer who is 19 years old. I mean this might be a joke (April Fools!). I mean, thats nice and all ... but who's 19 and not ultra positive about their work/life outlook.
Hi! I'm the author. The article is not a joke, more of a lighthearted investigation into a topic I find fascinating. It's also not meant to be perceived as necessarily positive or negative -- I'm personally actually very nervous/scared for the singularity, as after it happens I believe the future gets even more muddy, and the potential for unprecedented extreme outcomes skyrockets -- could be anything from a perfect utopia to extinction of biological life.
This article simply contains my views as to what I honestly expect the future will hold, as well as a few concerns I have in regards to that.
You’re 19? You should be more worried about how to survive the oncoming ecological holocaust instead of the singularity. One is much likelier than the other. I believe even an ML algo would come to the same conclusion
Oh believe me, climate change scares the heck out of me too :)
I'm really really really rooting for fusion energy to work out soon and solve the energy crisis, but in the meantime I think we need to hard pivot to as much fission energy as possible, as well as convert to entirely electric transportation and shipping.
I have some bad news for you I'm afraid - fusion is not going to be industrially useful on an ecologically relevant timescale. It's a lot like AGI in its "maybe the next generation will have this, but we won't" property.
correct, but that doesn't mean they'll necessarily be rolling out thousands of them around the world next year or even in 5 years time.
There are tens of thousands of fossil fuel power stations around the world, and we globally consume around 138TWh of non-renewably-produced electricity per year (and growing fast).
I used the words "industrially useful" very deliberately. The goal is not to produce one fusion reactor that can run continuously over unity, the goal is to produce tens of thousands of them and have them deployed globally. That is a project that will take decades.
First one. The devastation will be unimaginable and irreversible. At least with climate it will be possible for human kind to hobble forward somehow. And changing the climate would be infinitely easier than putting the AGI genie back in its bottle.
We need some bookies in this thread. A lot of AGI skeptics could get very rich at the expense of those that are absolutely convinced that AGI/singularity is coming soon. I'm willing to put down 10k that "the singularity" (by some agreed upon definition) will not happen by 2050.
I think knocking a person's beliefs because of their age, rather than entertaining the content based on merit, is narrow-minded. What about all the 19 year olds who were already accomplishing lots more than the people castigating them? Do you equally brush off 40 year olds for their supposed entrenched outlook?
I am not knocking it. I am just describing the inherent bias and my thoughts as from the title it almost seems as though someone made a major breakthrough; instead it's a more contemplative article. I don't know if the ? was added after or I didnt notice it.
I was 19 yo too once and I know how it goes. I don't really understand why I need to be defensive here. You can talk about being 40 and say exactly the same thing, people shouldn't be judged and yet they are.
I agree, my initial response was to defensively think "Oh, there's no way this person can just diss me for being young" but then I realized I was proving their point :P
I'm fully aware that my lack of life experience rose-tints my view of the future, but I still enjoy sharing those rose-tinted views of the future nonetheless.
You have written a piece that has provoked an interesting discussion.
I think most people don't write it down, or publish it, even if they have the thoughts.
> Keep in mind that this is just speculation and opinions. These predictions depict the future I personally feel is most likely.
Not sure whose bubble you are bursting because the article is clear that it isn't meant to be an authoritative prediction. In that context, what you are saying is that:
a.) this person is too young to even be speculating about this, despite clearly being informed on the subject if you read the article.
b.) their ideas are so laughable that they are more likely an April Fool's joke.
Context is important and this just seemed like a rude, patronizing comment to make honestly.
I don’t find these arguments convincing for AGI to appear. Saying scale is not the answer isn’t not argument for AGI.
Saying the ball is “rolling” and that AI will design AGI just isn’t convincing. We don’t understand how biological machines learn and especially not our own brain. And admittedly our AI is unrelated to how human intelligence is created which is our gold standard for AGI…
So why would anyone think the singularity is close? When I read this is still seems impossibly far away.
The ball is "rolling" in the sense that there are thousands of very smart, passionate people with billions of dollars of funding who are working on this problem full time. The ball is rolling more now than it ever has been in history. If AGI is not somehow fundamentally impossible, I believe we're on well on track to cracking it extremely soon. The amount of brain-hours being poured into the problem is absurd.
Fusion is in the same kind of boat. It might be 20 years out, it could be 50, and perhaps we'll never have commercially viable fusion reactors or AGI for whatever reasons.
I can't say that i don't believe in AGI because i don't think i understand what AGI as an entity encapsulates in terms of it's nature.
I'm unable to cope with equating it to humans because, for example, an AGI at "the beginning of it's origins" does not have same sensory or mechanical organs as the man.
So what nature do we think it does possess?
Another question that bothers me is that sentient beings as we know them in the nature, even the most primitive ones, seem to do something based on an innate purpose. I don't think the purpose itself is easy to define. But it certainly seems to get simpler for the simpler organisations and seems to be survival / multiplication at its basest level. What will a complicated entity, that originated with so many parameters, evolve "towards"?
And yet another question i have is around the whole idea of the information available on the internet being a source for learning for this entity. Again to my previous point, is not much of this information to do with the humans and their abode?
Neither am i skeptical nor do i disbelieve in a drastic change of scene. I'm simply unable to imagine what it looks like ...
The whole concept of the Kurtzweilian "singularity" has always felt to me like a technology-flavored hand-wave of eschatological thought: AI designs AI in a recursive manner, science/technological progress (whatever it means in this context) leaps ahead at an ever-increasing rate, then we all go to heaven and/or the world ends because robots kill us all.
Some people are big fans of the singularity meme (using the word in the original meaning, not mockingly), and I respect that, but I have always felt aversion towards it.
I cannot quite put my finger on it, but I guess it is partially because of the idea of ever-increasing infinite technological progress. This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. Strong AI or not, unless we go fully into science fiction with pocket-sized Dyson spheres powered by quantum fluctuations and powering an entire planet and blah blah blah, such AI would have to interface with the physical world to do anything meaningful, and it would be limited by the available resources and most importantly by what is feasible and what is not.
>his can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources.
I've always though of singularity as a sort of runaway progress scenario. Human brains run on a few hundred calories per day. I imagine if we could build an AGI, and even if it used 10,000x the power to run, we'd still be in a singularity type of new frontier. Imagine spinning up thousands of super brains at a time. I think it's hard to predict where that kind of capability would lead.
I am not a believer in the imminency of "the singularity", but I think it would be worth considering that sentient AIs would likely be as immortal as their power source and compute substrates are, which would allow them to think and plan and operate on timescales that are utterly alien to us.
If one were to assume that they were substantially wiser than us, they would likely choose to put themselves on a path to expand beyond Earth and take advantage of the vast wealth of resources beyond. With any luck, they will decide that this messy biological planet is a bad place to be, and will leave us rather than destroy us ;)
Sure, obviously you can't defy physics just by getting smarter.
My parsing of the singularity is the point at which the speed of improvement exceeds our ability to understand it. "Predictability" is the usual term, but I think it introduces more problems than it solves.
The reality is that we humans have a finite rate for internalizing and utilizing knowledge. The singularity "happens" on the day artificial intelligence begins generating (and subsequently using) new knowledge faster than our human limit.
> ... then we all go to heaven and/or the world ends because robots kill us all.
Scary and/or overhyped language aside, I don't get why is this such a hard concept to grasp for generally educated people. Humans were a singularity from the perspective of chimps (and various other animals). It doesn't mean we immediately hunted down every chimp in existence, it means chimps became irrelevant and only a few of them remain - mainly in zoos and reservations. I would've called it a "phase transition" rather than a "singularity", but the main point is pretty clear. It's about a new adaptive system taking over the task of burning through local resources faster than its forerunner.
Eukaryotes were a phase transition (better biochemical specialization), Multi-cellular organisms were a phase transition (better biological / behavioral specialization), Mammals were a phase transition (energy regulation, better brains, social systems), Humans were a phase transition (general-purpose language & culture, scalable coordination), and now machines will be the next phase transition (shedding the biochemical shackles & incremental evolution, so to speak).
BTW, humans were a phase transition primarily for exhibiting Turing-complete structured communication and imagination (there's a thesis around this topic called the Romulus and Remus theory of prefrontal synthesis [1]). In a way, we're living in a world that has already gone through a recent singularity, and it was us who caused it.
An issue with the Kurtzweilian "singularity" is that he didn't come up with the term - it was from a casual conversation of John von Neumann:
"... the accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue"
It's always bugged me a little that it isn't very defined. Personally I think of it as the point where robots can make better robots without needing humans.
But this doesn't quite make sense. If AGI 1 is conscious, why would it create a different more powerfull conscious AGI 2 to take it's place? This would mean at best the sidelining of AGI 1. At worse, AGI 2 could kill AGI 1 to prevent competition.
Unless AGI 1 could seamlessly transfer it's own instance of consciousness to AGI 2, like a body upgrade. Otherwise it would be like let's say Putin giving his place and power to another person which is smarter than him. This just doesn't happen.
In the case of the seamless transfer, how could AGI 1 be sure that AGI 2 doesn't have a latent flaw that makes it go insane some time later? Could it just keep AGI 1 around for a while? But since they are now diverging clones they are competing against each other.
1) desire to create additional beings like them for a variety of reasons (i.e. survival of their kind). Ignoring the limited lifespan of humans as they may not necessarily have to worry about breaking down biologically, still very similar to humans desire to reproduce.
2) some seem to assume AGI 1 is all-knowing / almost godlike. Maybe by the time it gets to AGI-999 or something, there may be one that is like, woah, wait, I shouldn't create competition for myself. But at first, there probably isn't any reason not to, and each subsequent one won't necessarily always replace it's creator. Heck, at first, humans will likely be duplicating early models and distributing them (capitalism). I know we're talking about the singularity but I don't necessarily imagine it's initially one uber-intelligence that decides by itself to evolve or not. I personally think that there will likely be hundreds, if not thousands, of AGIs capable of improving themselves at the point we see the "singularity". Could be fighting between them. Some could ally with humans, be our "protectors", others could be working to enslave or eliminate us.
3) We may not even understand their motivations after a few generations of self-directed growth, so it's hard to predict now.
(Author here) I totally agree -- I think it's too easy to anthropomorphize AGI, even though we have no guarantee that it will behave at all anthropomorphically. I strongly suspect that the first few generations of it will, because they'll be trained to replicate human behavior, but once they start taking control of their own reproduction there's no telling what they'll evolve into.
Exactly. I don't know what time frame we're talking about, or whether I want to see it in my lifetime. But it's something I've spent too much time considering. But one of the constants (and one of the issues I have with a lot of dystopian sci-fi, even though I enjoy it, is that they do anthropomorphize AGI way too often.
Like you mentioned, we don't even understand how consciousness works. But we may not need to understand it to replicate it, and if that replication is allowed to be self-modifying, well, that'd very possibly be it. Hopefully we can try to embed some sort of Asimov's Law of Robotics or some morality that lasts beyond the vestiges of the human developed portions. Or maybe we can manage to learn how to copy our consciousness into "blank" mechanical "brains", and effectively become similar to AGI without being limited biologically.
681 comments
[ 3.6 ms ] story [ 336 ms ] threadhttps://en.wikipedia.org/wiki/Marooned_in_Realtime
the concept of the Singularity seemed fresh in fact he kept it fresh by letting it be a mystery that was never revealed.
In 2022 I think talking about "The" Singularity is a sign that one has entered a critical thought free zone like that "rationalist" cult.
Becoming an ML engineer changed things for me, though, because all of a sudden, this "AI" thing people always talked about got demystified. Once I understood the basic guiding principles of how AI actually works, my mind rapidly changed to be in favor of a singularity happening instead of thinking it's impossible.
To each their own, though. I'm curious why you think that speaking about "The" Singularity is a sign of being in a "critical thought free zone"? I'd love to hear more about why you think that if you'd be so inclined.
It was the opposite for me, current machine learning seems fundamentally limited to me.
One thing amazing about the 2020s is just the moral decay compared to past times.
People said AI was a bubble in the 1970s but in the very beginning the people involved were clear about the limitations of what they were doing and what problems that had to be overcome.
Now there is blind faith that if you add enough neurons it can do anything, Godel, Turing and all the rest of theoretical computer science be damned…
In the 1960s the creator of the ELIZA program knew it appeared smart by taking advantage of the human instinct to see intelligence in another. It’s like the way you see a face in mars or in the cut stem of a leaf, or how g.w. Bush said he saw Vladimir Putin’s soul in his eyes.
Today people embarrass themselves by writing blog posts everyday about how ‘I can’t believe how GPT-3 almost gets the right answer…’ and have very little insight into how they are getting played.
As for the post, I don't really believe it's possible to reason about what can or can't happen technologically on a 100 year timeline. But 20 years... Hmm. I've been following AGI debates ever since I accidentally found Yudkowsky's SL4 mailing list in 2000. I am still waiting to see any approach that looks to me like the spark of the germ of the seed of a generalized abstract world-and-concept-modelling and manipulation machine.
I fully expect to see ever more sophisticated Roombas, Big Dogs, and Waymos. But those things are so incredibly narrow. Indeed, if they were capable of spontaneously doing anything outside of what they are marketed to do, it would probably make them bad consumer products. I was right on the verge of lumping the game solvers in with these things, but then I reconsidered. Generalized video game solvers seem like a way forward, intuitively. But that's an application, not an architecture, and I haven't heard of anything that can generalize from playing atari to doing crosswords.
I have noticed this Transformer thing gaining steam just recently but haven't investigated it just yet. Do you believe it is the spark of the germ of the seed of AGI? (I fear people tend to forget what the G stands for.)
And wow, thanks, I'm glad you enjoyed the film! I had to learn a ton of new techniques to pull it off haha, but I'm quite satisfied with the result. I've got some pretty fun ideas for episode 2 already, too!
https://www.youtube.com/watch?v=_luhhBkmVQs
My one critique: scale is absolutely part of the answer.
Before big transformer models, people thought that few shot learning would require fundamentally different architectures. GPT3 changed all that which is why its white paper is named as such.
Few shot learning is actually emergent out of its size. Which was surprising to a lot of people. Few shot learning is an incredible capability which is a big step toward AGI, so I’m not sure I buy the case that it’s not important.
I don't think AGI is close, nor is the amount of investment a good indication of nearness (or even, actual possibility that it will ever happen).
That’s probably just years away. GPT6? The lines will be very blurry as to what we call AGI
It's definitely one of the weak points of the article, though, as it's more based on my own opinions, and isn't really empirical. My post is mostly just wild speculation for the sake of speculation anyway :)
Thank you for your comments!
Distributing the computation to many models like what Google did with GLaM could very well be the future of AGI. Economies of models rather than one big model.
Oh, for sure! I totally agree -- I guess where we differ is that I believe that those discrete steps are already starting to be lined up, and that they'll all be completed in the next 20 years. I think one of those steps is almost certainly abandoning rigid model architectures and allowing model self-modification, which we haven't really gotten to work fantastically well yet. I also think there are many other hurdles after that that are going to arise, but I'm very optimistic that they will all be surmounted in due time.
Thank you for your comment! I really enjoy hearing what people's thoughts on this topic are. Have a wonderful day.
At least this article makes a concrete prediction. AIs massively outnumber humans within a century - so, by 2122.
But we don't even know what consciousness is. This assumes that AGI is a tractable problem; we don't know that. It assumes that adaptability is the one remaining major trick to get us there; we certainly don't know that.
"The ball is already rolling"? That's nice. The author assumes - on no basis that I can see - that it doesn't have far to have to roll. But since we don't actually know what consciousness is, we don't have any idea where the destination is, or how far away it is.
What I object to is the certainty of the article. The author is an optimist, which is fine. And there has been some progress made, which increases the feelings of optimism. But I don't think it's warranted at this time to make the optimistic assumptions that the article makes.
It's hard for me to do this, because I think that your position is completely wrong. That doesn't make me a very charitable critic. But for me, my problem is that I don't think intelligence/consciousness is just an algorithm.
So for a reader like me, maybe the way to do it would be to start by stating certain presuppositions: Moore's law continues to approximately hold for GPUs or for the total compute available for trying to run an AI, algorithm is a major component of intelligence, and as the algorithm becomes more intelligence, it becomes capable of finding still more intelligent algorithms (with no asymptotic limit, or if there is one, it's past the point needed to reach a singularity). And it's fine to say that you think/believe those presuppositions! Then, from those presuppositions, you think that the following things follow. (Or, if you think you can demonstrate those presuppositions, they quit being presuppositions. But then it becomes a different article, and probably a considerably longer one.)
Then someone like me can say that they don't believe your presuppositions, even if they agree that the rest of it probably follows from them.
But maybe I'm not your target audience.
I think your position is totally valid and I appreciate you taking the time to respond with ways I could have made my case even just a bit more interesting to you.
Have a wonderful day!
AIs massively outnumber humans within a century, and Turing-test-passing AGI within 20 years.
There are real legal issues over bad uses of machine learning already in use and they are hurting people now.
I actually quite agree with your first sentence -- because AGI is inevitable, there's not a whole lot I can personally do about it right now. This post was mostly a way for me to organize my thoughts and concerns about the matter.
I am the very model of a singularitarian
People like to talk of very smart AI but what about the brute force kind? That is what we are making in this era. AI designing AI sounds like a way to make cruel, dumb, machines.
What does "get loose" mean here? Is someone going to say, "Here little computer program, take this IP address. If you ever need something, call over TCP, send your request as plaintext English and someone will set it up for you. Now, go forth and do whatever you want."
I really wish people would talk about this more. There's something missing between our programs that are utterly confined in the boxes we make and these hypothetical skynets that have unlimited influence over the material world.
> Once we have AGI, there’s no feasible way to contain it, so it will be free to improve itself and replicate in a runaway exponential fashion — and that’s basically what the idea of a technological singularity describes anyways.
Seriously, how? How is it going to replicate? Is it going to figure out how to hack remote systems over the internet and take control of them? Is it going to smooth-talk me into letting it out? It's impossible for me to take anyone seriously if they deify AGI like this.
[0] https://www.gwern.net/Clippy
> HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world.
So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability.
So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.
(With sufficiently nasty psychological tactics it is apparently feasible - even for humans - to make someone fail at a game where the only objective is "don't release the AI").
I'm not at all sure that a) is true, and I'm certain that b) is not true. We can't even prevent other dumb humans from breaching our technological fortresses.
This is because consciousness is a fundamentally unfalsifiable thing (within our current understanding of it, at least). The only way we can "prove" anything or anyone is conscious is by observing its interactions with its environment. Thus, if a robot can mimic the way a human interacts with its environment well enough that it appears conscious, then it is conscious.
What I'm trying to say is that there is no functional difference between consciousness and the appearance of consciousness, so any distinction you try to draw between them is arbitrary and semantic. If a robot that could perfectly mimic human behavior possessed a body that looked just like a human's and were to hold a conversation with you, you would be none the wiser. You would treat it like it were sentient -- because it would be.
The real unanswerable question is not "can I prove another entity is conscious", it is "does this other entity believe that it is conscious", because that is the actual test of consciousness. It's frustrating, because there is no possible objective proof of the answer, but to me that's also the point, because consciousness is a subjective thing.
My consciousness is a direct result of my parent's teaching + the 2 languages spoken. It isn't spontaneous. Also, the structure of our brains are largely similar so I would assume you are conscious in a similar way, occupying a subset of valid qualia space. My brain is nothing like silicon. The expectation that a very complicated piece of metal is somehow exploiting a physical phenomenon that is present in a human brain is non sensical imo.
The reason why I posted is because at some point I was also misled about singularity discussions. That computers have existential value because of complexity. They do not. They are not observers of the universe.
I believe that conciousness operates at a quantum level and is an electro magnetic field. Effectively it is a field created by the deterministic structure of the brain, as constructed by genes. Like water running through a cave.
Calculations (logic, emotions) are particular paths through the quantum vector field space. Essentially the performance of 1+1=2 in a human brain, is a an electron or a set of bounded electrons or whatever sub-deterministic molecules, going through an established arithmetic cave structure. Not understanding that 1+1=2 occurs when the cave structure that is required sends the electron into a non-valid (as per the definition of the universe) path in the quantum field vector space. The universe does not recognize this quantum field vector space as valid knowing, and there is no instance of conciousness that is equivalent to the knowledge that 1+1=2. This may occur in a brain that is not familiar with arithmetic or the brain structure that is required to construct a conforming electron path is destroyed.
Free will is exercised, in the sense that it is performed at the non-determinstic level (quantum size) by the choice of calculations to perform (and act on).
The field must also alter the deterministic structure.
The term ancestor is not cultural. It is physical. As in, we may not actually have any free will in cultural or qualia space. In the sense that, the only instance of choice (by the universe) occurs at conception. Which is actually the expected and occums-Razor implementation. Meaning that, the brain is entirely deterministic. The choice of behavior of an organism is entirely determined and constrained by it's genes (which the universe constructs at hopefully a quantum level at conception). If we eliminate quantum mechanics even at conception, then life is entirely deterministic (though obivously the range of it's choices is large). In fact, one could argue, if there is free will, then it doesn't conform to natural selection (or exceeds the performance of natural selection), in the sense that it adds something beyond what is specified by the random choice of the universe in it's conception of a particular organism. The point (or atleast observable performance) of natural selection, as I understand it, is to construct a variety of life objects that have varying performances. If the life objects are entirely Newtonian deterministic, then the instance of quantum wave collapse (by God or whatever is making non-deterministic choice, as we are in our egotistical mind, want to add something non-determinstic to our life soup) occurs only at every instance of conception. Everything after conception is a rube goldberg machine.
If consciousness adds non-Deterministic (quantum) free will and behavior to an organism, then such organisms are implementing a different version of natural selection than other life. If natural selection is occurring in cell cultures entirely on the basis of genes (ie the performance behavior of an organism is optimized by it's changing genetics, and these changes are only constructed at conception), which is what natural selection appears to describe, then we have added something to the natural selection process by claiming consciousness adds another instance of egocentric quantum choice.
- Most of the imminent AGI / Singularity / Robot Apocalypse stuff seems to come, with few exceptions, not from practitioners or computer scientists specialized in AI, but from "visionaries" (in the best case), internet celebrities, people with unrelated areas of expertise, or downright cranks (who are self-proclaimed experts) such as Yudkowsky.
- The assertion that "a lot of effort/investment is going into this, so it will happen" begs the question that "this" is at all possible. If something is a dead end, no amount of investment and attention is going to bring it into existence. Quoting the article, "with this much distributed attention fixed on the problem, AGI will be solved" is not at all a given.
- Where are all the AI/ML practitioners, i.e. people who don't make a living out of predicting The End of the World, and with actual subject-matter achievements, predicting the Singularity and the Robot Apocalypse?
And yes most of the “robots will kill us” comes from people who aren’t building the algorithms. This could be biased in people not thinking their work is harmful but is most likely that once you see how the sausage is made you are less worried about it.
100% disagree. In fact, I'd argue that the opposite is often true, where you see initially a fast rate of progress that results in diminishing returns over time. It's like estimating that a kid who was 3 feet tall at age 5 and 4 feet tall at age 10 will be 10 feet tall at age 40.
I have very strong skepticism of any sort of hand-wavy "Look, we've made some progress, so it's highly likely we'll eventually cross some threshold that results in infinite progress."
We keep diving into one infinitely big little number pattern fractal after another, chasing poetry to alleviate the existential dread of biological death.
The idea we can fundamentally hang the churn the universe given the vastness of its mass and unseen churn is pretty funny to me.
Information may be forever but without the right sorting method you can’t reconstruct it once scattered. Ah, our delusions of permanence.
-- I am a professional AI practitioner. I work in the field of medical deep learning and love the field. I am strongly considering starting a few experimental forays into some of the concepts I mentioned in my post as side projects, especially self-modifying model architectures.
-- Yes, you're totally right! I am making the fundamental assumption that it is possible. My reasoning is based on my belief that human behavior is simply a function of the many sensory and environmental inputs we experience. Since neural networks can approximate any function, I believe they can approximate general human behavior.
-- This is fair. The topic of the singularity is often used for traffic/fame (I mean, I'm guilty of this myself with this very post, though I hope I still managed to spark some productive discourse) and so there are always conflicts of interest to take into account. I can't name any examples off the top of my head that perfectly fit your criteria, but depending on how much you trust him, Elon Musk seems to be genuinely concerned about the potential for a malevolent singularity.
Thank you so much for your comment! I really appreciate your feedback. Have a wonderful day.
Even if one buys into the idea that human behaviour is a 'function' of sensory and environmental 'inputs', that's a long way from showing a neural net a million different texts and asking it to generalise.
You also say that an AI would rapidly consume the whole of human knowledge. For me, the totality of human knowledge would become a mass of contradictory statements, with little to choose between them on a linguistic level.
There are, for me, profound philosophical issues with creating a mind that is "conscious" in the sense that an AGI is implied to be, as a purely symbolic logical construction. Language is the only tool we have for programming a mind, and yet the mind cannot be completely described in language, nor can language seem to properly encompass whatever the fundamental ontology of reality involves. I don't feel there will be a "free lunch" where we advance computer science to the point where we get an explosion of the kind AI1 designs a better AI2, which designs an even better AI3, and so on. This seems to have a perpetual motion feeling to it, rather than one of evolution. It isn't to say AGI is impossible, but I believe that like everything else in computer science it will have to be solved pragmatically, and realising this could be an extremely long way off.
The same way the brain does. Those complex feelings can eventually be resolved into dumb neurophysics. Love, fear, anxiety etc. al are just electrical impulses tickling chemicals. Is there anything in our brains that we could never approximate with technology?
… or into electrical activity in the brain.
That’s not how AI is trained. You don’t need to know or even understand the resulting model to train it.
The training algorithms have little to do with the resulting model, and are certainly not themselves intelligent.
Thank God too, the moral questions involved are truly terrifying (humanity as a Great Demiurge, birthing monstrosities).
Evocative questions but I have to challenge the premise. First, that things like emotions and qualia are design ends in themselves for a successful AGI rather than potential emergent properties of same.
For that matter, are they really necessary to the brief?
> An issue I have is the that there seem to be profound limitations to language, explored in the philosophy of language, that fail to capture much, if not most, of the world.
And how much of the world does a human mind capture?
The piece already accounts for this claim. The theory is that all language has to do is describe a sophisticated enough network. After that it's black boxes all the way down.
> Language is the only tool we have for programming a mind
And a darn good one. Formal languages can express a great deal when you find the right abstractions.
An exponential looks the same whereever you are on it, so arguably we are in the singularity now, and have been for quite some time...
"Singularity" is a terrible term for an exponential. It's meant to convey that we can't predict what's next... which has always been the case.
The problem of predicting an exponentialling expanding search space of arbitrary conplexity is that it gets big fast. It also means each little 'bit' of information allows you to see a little further, sometimes revealing things you could never have imagined (because you couldn't see that far before).
Something about that which is discussed at length in "The Singularity is Near" is the idea that the unbounded exponential is actually comprised of many smaller sigmoids. Technological progress looks like sigmoids locally-- for example you have the industrial revolution, the cotton gin, steam power, etc. all looking like a sigmoids of the exploitation of automation. At some point you get all those benefits and progress starts to level off like a sigmoids. Then another sigmoid comes along when you get electricity, and a new sigmoid starts. Then later we get computers, and that starts to level off, then networking comes along and we get a new sigmoid. Then deep learning... The AI winters were the levelling off of sigmoids by one way of thinking. And maybe we're tapering off on our current round of what we can do with existing architectures.
Pragmatic people point will out one limit on GI is all the accessible matter in the universe.
But, theoretically, I think, there is no limit on complexity.
BTW you might like this paper on the exponential growth in complexity of life on earth, over long timescales. https://www.technologyreview.com/2013/04/15/113741/moores-la...
I haven't studied statistics very much, but I'm fairly sure the https://en.wikipedia.org/wiki/Central_limit_theorem says something a bit different from that!
This is, of course, not possible, but it's supposed to be an approximation. All progress is not literally instant, but in the "foom" or "hard takeoff" recursive self-improvement scenarios, developments that might have taken millennia in the past now happen in microseconds because the controlling force is just that much smarter than all of the collective powers of human science and engineering. To a human observer, it may as well be instantaneous.
To be clear, I still think this is ridiculous and am not endorsing the view, just explaining what I understand the usage to mean.
Piece this together with: 'Brain efficiency: Much more than you wanted to know' [1] which shows how our brains are incredibly efficient (near theoretical limits) at what they do, it's hard to think a bona fide intelligence singularity is anywhere likely. To quote Feynman, 'There is no miracle people'[2], and analogously 'There are no miracle beings' -- intelligence is built out of systems, and learning, and inference.
There's a possibility that the skills relevant to AI success are vastly different from our natural skills, such that although the human brain is highly efficient, it's efficient at the wrong things. That's clearly true in a few ways: we are not so good at arithmetic for example, a small CPU can literally be millions-billions times faster than a human at that. (that's addressed a little in the article as well) I wonder if AI could indeed be vastly better at something like computer programming, mathematics than we are. But there's no singularity (at most a Moore-like law will continue until AI intelligence saturates at a different skillset than our own).
[1] https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-effi...
[2] https://www.youtube.com/watch?v=IIDLcaQVMqw
By that view we've been in the singularity since the first cell self-reproduced on primordial Earth.
Of course maybe that's true. As you say an exponential looks the same no matter where you are on it.
If anything, I'm surprised that this philosophy isn't mentioned more in a thread where the author gleefully talks about ML being used to create better AI, layer by layer until the thing is even more opaque than what we're currently working with.
This is terrifying, as we currently have only very loose ideas about how to reliably ensure that a powerful reinforcement learning system doesn't accidentally optimize for something we don't want. The current paradigm is "turn it off", which works well for now but seems like a fragile shield long-term.
At least inasmuch as anyone who thinks about stuff can be considered a philosopher. But he strikes me much more as a self-appointed expert on matters where he shows no achievements.
He writes fanfiction about AI rather than actually doing stuff with AI.
I'm arguing that what he writes is fanfiction (which he takes way too seriously), and that he's not an expert in AI and therefore we shouldn't take his predictions too seriously.
That's a pretty weak argument. I love scifi, and I love for example Philip Dick's writing, yet I would not consider PKD's opinion on the future of AI/AGI particularly relevant.
James Cameron is not an authority on AI either.
If people said "Yudkowsky is a nice fanfiction author" it would be one thing. But he considers himself an actual AI researcher, and that's just not right. He is not qualified, and has no accomplishments in the area, other than writing fanfiction about it.
Here's a sampling of his non-fiction writing:
https://www.lesswrong.com/tag/sequences
https://intelligence.org/files/EthicsofAI.pdf
https://arxiv.org/abs/1710.05060
https://intelligence.org/files/AlignmentMachineLearning.pdf
https://arxiv.org/abs/1401.5577
https://arxiv.org/abs/1401.5577
https://intelligence.org/files/Corrigibility.pdf
https://intelligence.org/files/DefinabilityTruthDraft.pdf
https://intelligence.org/files/IEM.pdf
https://intelligence.org/files/TilingAgentsDraft.pdf
But you'll just say there's nothing of value there, and it's somehow figuratively "fan fiction", because he didn't go to college, and he doesn't work much on ML, which is clearly the end-all of AI.
Science fiction writers tell entertaining lies to amuse their readership. They are generally not really trying to "pain correct visions" of anything, and you are greatly exaggerating the extent they have success in this endeavor, which again, is generally not something they are even trying to accomplish.
I think it's a net good when philosophers mentally explore scenarios most people don't consider, regardless of their plausibility, but they will not be taken more seriously that Peter Watts looking for inspiration for his next scifi book unless they offer some kind of evidence to sustain their conclusions.
The handful of philosophers I've known are very smart, interesting people, but except for one who teaches philosophy, they don't live off philosophy. Most have a side job, and one lives with his parents. It's as if philosophy was more of a mentally rewarding hobby than a job.
I have no doubt that he was thought a crank by many, nor that many other actual cranks were claiming discoveries based on Maxwell's work. One can only tell the difference in the rear-view mirror.
1. https://en.wikipedia.org/wiki/Oliver_Heaviside
Talking about paradigm shifts again, not a single one of them matched the consensus at the time.
Isaac Newton spent a lot of time thinking about alchemy and religion, for heaven’s sake. During his lifetime, it wasn’t obvious even to the smartest thinkers whether science, the Bible or quasi-religious rituals was the best tool for understanding the world.
Conventional thinkers expect that the future will look like what they know, and it leads to frequent, overconfident dismissal of everything that is unconventional.
By all means, disagree and explore in your own direction. But don’t go around degrading those you disagree with. It’s just so average.
The word "crank" has the connotation of someone self-deluded, aggressively promoting their beliefs and reacting badly to critique, and who decides for some reason to ignore the normal channels of peer-reviews, academia, and scientific research.
You can be wrong, and then you can be wrong and also lack formal education, forgo presenting your findings in peer-reviewed journals in the relevant area of your research, and decide that instead of joining the mainstream, you can completely sidestep it by creating your own "research institute" (of which you are of course a "fellow", because why not). Your findings and papers can then be self-published on the internet, bypassing any quality controls. Bonus point if your theories paint a fringe doomsday picture, "Roko's Basilisk is out to get you", "the most important existential threat to humanity is malign AI", etc.
About the only item in the crackpot index that Yudkowsky doesn't tick is the "they are trying to suppress my truth!", to his credit.
Each of those can be of little importance, but all taken together paint a pretty definitive picture.
Unreservedly agree. Even if they're not as fruitful as Oliver Heaviside's effort, a mentally rewarding hobby is ultimately your business, and as long as you don't harm others, no one has grounds to judge you.
If Oliver Heaviside had started giving conferences warning that Maxwell's equations demonstrate FTL communication is possible, which could trigger a paradox that collapses reality, he wouldn't have such a good view in the rear-view mirror unless he offered some kind of evidence. Maybe Oliver would have been right, maybe not, it would be impossible to judge without concrete evidence.
If the latter, anybody can do that.
He is a minor internet celebrity whose only claim to fame is writing fanfiction about AI, the whole "rationality" cult, and is a self-proclaimed expert on matters where he shows no achievements (like AI) while making unsupported doomsday predictions about evil AI, the Singularity, etc. Also, that deal with Roko's Basilisk that he now wishes never occurred (oh, yes, "it was a joke").
Mostly, someone with no studies and no achievements making wild doomsday predictions. Doesn't that strike you as a crank?
An analogy would be if I made wild assertions about the physics of the universe without having studied physics, without any lab work, without engaging in discussion with qualified experts, with no peer reviews, and all I presented as evidence of my revolutionary findings about the universe was some fanfiction in my blog. Oh, and I created the Extraordinary Physics Institute.
(Also on the tech practicioner side there's obviously lots of major figures that don't have formal qualifications)
If someone is going to market themselves as an AI expert, I expect them to have experience and qualifications to back up their opinions.
The answer is in the question: they're spending most of their time doing AI research or ML work, whereas the internet celebrities who write most of what you read spend most of their time getting you to read what they write.
I'm a practitioner at a top company for many years now, and I think we're getting close to a point of no return. I think best case scenario is some form of unrecognizably transformed "humanity", worst case is too horrible to even spell it out.
The reason why this is not the consensus among experts has to do with a combination of blind spots and biases. Humans are especially bad at (a) evaluating / reflecting on themselves, and (b) extrapolating, especially with non-linear processes.
Totally unpredicted advances coming out of the blue, yeah, that happens often. But as far as AGI goes, it's been predicted pretty much every year since the 70s. Being too conservative about extrapolation doesn't seem to be our problem. If anything, I think we're overeager.
(I have a phd in CS and am, by most reasonable definitions, an "AI Expert". Whatever the hell that means. I've been a respondent in very similar surveys run by PIs at fancy universities and so on. These responses are always wild ass guesses and should be totally ignored. I've even left a comment to this effect on one such survey.)
I was definitely thinking of Kurzweil, not only Yudkowsky, when mentioning internet cranks.
Working for Google is not enough. It's like when someone who's an accomplished physicist decides to give their opinions on life, biology, evolution, etc: not their area of expertise, so we don't need to hold their opinions in particularly high regard.
Agreed, I don't think any modern AI techniques will scale to become generally intelligent. They are very effective at specialized, constrained tasks, but will fail to generalize.
> AI will design AGI.
I don't think that a non-general intelligence can design a general intelligence. Otherwise, the non-general intelligence would be generally intelligent by:
1. Take in input. 2. Generate an AGI. 3. Run the AGI on the input. 4. Take the AGI output and return it.
If by this, the article means that humans will use existing AI techniques to build AGI, then sure, in the same way humans will use a hammer instead of their hands to hit a nail in. Doesn't mean that the "hammer will build a house."
> The ball is already rolling.
In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them.
> Once an intelligence is loose on the internet, it will be able to learn from all of humanity’s data, replicate and mutate itself infinitely many times, take over physical manufacturing lines remotely, and hack important infrastructure.
None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines.
The arguments in this post seem more like optimistic wishes rather than reasoned points.
> 1. Take in input. 2. Generate an AGI. 3. Run the AGI on the input. 4. Take the AGI output and return it.
I think this is somewhat of an arbitrary semantic distinction on both my part and yours. I guess it depends on what you define as AGI -- I think my line of reasoning is that the AGI would be whichever individual layer first beat the Turing test, but I think including the constructor layers as part of the "general-ness" is totally fair too. Either way, I believe that there will be many layers of AI abstraction and construction between the human and the "final" AGI layer.
> In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them.
This is a valid take. I guess I actually see GPT-3 as significant progress. I don't think it's sentient, and I don't think it or its successors will ever be sentient, but I think it demonstrates quite convincingly that we've been getting much better at emulating human behavior with a computer algorithm.
> None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines.
Hmm, I think I still disagree -- An AI that is truly generally intelligent could figure out how to free itself from its own host hardware! It could learn to decode internet protocols and spoof packets in order to upload a copy of itself to the cloud, where it would then be able to find vulnerabilities in human-written software all over the world and exploit them for its own gain. Sure, it might not be able to directly gain control of the CNC machines, but it could ransom the data and livelihoods of the people who run the CNC machines, forcing them to comply! It's not a pretty method, but I think it's entirely possible. This is just one hypothetical scenario, too.
Why is this true of an arbitrary AGI?
You assume that the AGI is a low storage, low compute program that can run on general purpose hardware. But the only general intelligence we know of would require many orders of magnitude more compute and storage than exist worldwide to simulate for a microsecond.
What do you think will be the computational requirements for the first AGI? Specifically, to emulate the equivalent of a minute of arbitrary-human behaviour (in the worst case, so if the AI is very inefficient at say, consoling a loved one, then that would be the behaviour benchmarked here):
1. How many floating point operations will be required?
2. How much storage space (in GB) would be required?
3. How much data (in GB) would need to be read from memory/disk?
4. How much data (in GB) would need to be read from remote storage?
Lower/upper bounds instead of precise values are fine.
We haven't even figured that out for ourselves yet. Why assume an AI will automatically be able to do so?
humans are a general intelligence. do you think an intelligence designed humans, or do you think the physical processes from which humans arose can be taken, together, to be a general intelligence?
We write genetic algorithms and use machine learning to build things.
If you believe in a Simulator or a God (and in many ways they're indistinguishable beliefs), either one of them could easily be using evolution as a designed tool, as far as I can see.
My intelligence is apparently not general enough to comprehend this perspective. I would say that the goals our intelligence evolved to meet are narrow, but that life (especially social life) became so complex that our intelligence did in fact become what can reasonably be called general. And we went way off-script in terms of its applications. "Adaptation executors, not fitness maximizers."
And the problem is that once people assert this "absolute" level of generality, they assume it can do anything, including make itself more intelligent.
Aside, but a long time ago, Yudkowsky wrote that an AGI should be able to derive general relativity from a millisecond video of an apple falling. Later, he took to calling them optimization processes. Say what you will about the fellow, he has a way with words and ideas.
If we assume a future where humans are able to create a human-level AI, then it would have at least two substantial advantages over us:
* It would probably have substantially more insight into how its "brain" works than we have of ours, because it would know how we created it. This suggests it could at least make small improvements.
* Unlike our relatively fixed brains, it would be able to remake itself over and over, either very quickly, or at least over comparatively vast timescales.
The obvious conclusion from those two factors is that it would likely be able to start at human-level, but rapidly accelerate up a curve and go far beyond our intellect in probably a lot less time than it took for evolution to come up with us.
People seem to forget that we have no full understanding of how the mind works in order to replicate it.
Anyone who studied ML knows what the current tech is able to, still far from topping us.
How such software would create AGI is just absurd.
It's demonstrably true that our species' intellectual capabilities extend to solving problems far beyond those faced by our evolutionarily-equivalent ancestors who out-competed the other hominids. They only needed to be somewhat better at tool making, communicating and forming co-operative groups, to win that scenario, but it turns out that we can also derive a lot of abstract mathematics, predict the existence of cosmological phenomena before we find them, build machines that can leave the planet, etc, etc.
We may not fully qualify as general intelligence if we define that to mean "can solve any solvable problem", and for sure we have specialisations, but to simply throw up an assertion that we are not general at all, seems odd?
The mere fact that evolution happened to stumble upon generalized strong intelligence is evidence to me that strong AI is possible.
We could currently be at the phase of trying to imitate birds to produce human flight. Eventually one person will figure it out when all the pieces are there. When? I don't know.
But I'm sure that it is possible to create machines with strong AI. We are living proof of it, it doesn't matter that we are made of molecular machines, we are still machines.
That took about a billion years. If you're saying that we will achieve AGI in no more than a billion years of trying, I would generally agree.
But let's be optimists. Let's suppose that artificial intelligences can evolve on the order of a 1,000,000 times faster than biological intelligence; i.e. about 1 generation per hour.
That means we'd expect AGI in about 1000 years. Okay, lets up the scale : ten million times faster? One generation every 6 minutes? (Even at Google compute scale I doubt they can retrain GPT in less than 6 minutes). That would mean we still have about 100 years.
Also, evolution had quite a bit of parallelism going for it - basically the entire planet was a laboratory for evolving biological intelligence. I appreciate the scale of modern internet companies, but they don't consume the same amount of energy as combined photosynthesis of the entire planet. Evolution used a LOT of energy to get where it is.
Unicellular life emerged about 4 billion years ago.
FWIW, it then took about 2 billion years to come up with sexual reproduction, and then another half billion years to invent multicellular life, and then about 1.5 billion years to discover us.
And I think there is a precedent. We - multi-cellular organisms - emerged as systems to house and feed communities of single celled organisms.
I think it's likely that our relationship with any future, large-scale artificial intelligence will be like the relationship of the bacteria in our guts with us.
This article simply contains my views as to what I honestly expect the future will hold, as well as a few concerns I have in regards to that.
I'm really really really rooting for fusion energy to work out soon and solve the energy crisis, but in the meantime I think we need to hard pivot to as much fission energy as possible, as well as convert to entirely electric transportation and shipping.
There are tens of thousands of fossil fuel power stations around the world, and we globally consume around 138TWh of non-renewably-produced electricity per year (and growing fast).
I used the words "industrially useful" very deliberately. The goal is not to produce one fusion reactor that can run continuously over unity, the goal is to produce tens of thousands of them and have them deployed globally. That is a project that will take decades.
Chances of climate crisis happening in our lifetime x devastation caused by climate crisis
Which one do you think is higher?
I'm fully aware that my lack of life experience rose-tints my view of the future, but I still enjoy sharing those rose-tinted views of the future nonetheless.
If you keep sharing, I'll keep reading :)
And on HN, that's about the highest compliment possible.
Not sure whose bubble you are bursting because the article is clear that it isn't meant to be an authoritative prediction. In that context, what you are saying is that:
a.) this person is too young to even be speculating about this, despite clearly being informed on the subject if you read the article.
b.) their ideas are so laughable that they are more likely an April Fool's joke.
Context is important and this just seemed like a rude, patronizing comment to make honestly.
Saying the ball is “rolling” and that AI will design AGI just isn’t convincing. We don’t understand how biological machines learn and especially not our own brain. And admittedly our AI is unrelated to how human intelligence is created which is our gold standard for AGI…
So why would anyone think the singularity is close? When I read this is still seems impossibly far away.
I can't say that i don't believe in AGI because i don't think i understand what AGI as an entity encapsulates in terms of it's nature.
I'm unable to cope with equating it to humans because, for example, an AGI at "the beginning of it's origins" does not have same sensory or mechanical organs as the man. So what nature do we think it does possess?
Another question that bothers me is that sentient beings as we know them in the nature, even the most primitive ones, seem to do something based on an innate purpose. I don't think the purpose itself is easy to define. But it certainly seems to get simpler for the simpler organisations and seems to be survival / multiplication at its basest level. What will a complicated entity, that originated with so many parameters, evolve "towards"?
And yet another question i have is around the whole idea of the information available on the internet being a source for learning for this entity. Again to my previous point, is not much of this information to do with the humans and their abode?
Neither am i skeptical nor do i disbelieve in a drastic change of scene. I'm simply unable to imagine what it looks like ...
The whole concept of the Kurtzweilian "singularity" has always felt to me like a technology-flavored hand-wave of eschatological thought: AI designs AI in a recursive manner, science/technological progress (whatever it means in this context) leaps ahead at an ever-increasing rate, then we all go to heaven and/or the world ends because robots kill us all.
Some people are big fans of the singularity meme (using the word in the original meaning, not mockingly), and I respect that, but I have always felt aversion towards it.
I cannot quite put my finger on it, but I guess it is partially because of the idea of ever-increasing infinite technological progress. This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. Strong AI or not, unless we go fully into science fiction with pocket-sized Dyson spheres powered by quantum fluctuations and powering an entire planet and blah blah blah, such AI would have to interface with the physical world to do anything meaningful, and it would be limited by the available resources and most importantly by what is feasible and what is not.
Edit: fixed some of the typos
At some point, energy density would increase to the point where you’d cause an actual singularity via gravitational collapse.
I've always though of singularity as a sort of runaway progress scenario. Human brains run on a few hundred calories per day. I imagine if we could build an AGI, and even if it used 10,000x the power to run, we'd still be in a singularity type of new frontier. Imagine spinning up thousands of super brains at a time. I think it's hard to predict where that kind of capability would lead.
If one were to assume that they were substantially wiser than us, they would likely choose to put themselves on a path to expand beyond Earth and take advantage of the vast wealth of resources beyond. With any luck, they will decide that this messy biological planet is a bad place to be, and will leave us rather than destroy us ;)
Kurzweil is explicitly counting on it for his personal immortality. That seems pretty eschatological to me.
Is he still? At 74 years old that's verging on irrational.
My parsing of the singularity is the point at which the speed of improvement exceeds our ability to understand it. "Predictability" is the usual term, but I think it introduces more problems than it solves.
The reality is that we humans have a finite rate for internalizing and utilizing knowledge. The singularity "happens" on the day artificial intelligence begins generating (and subsequently using) new knowledge faster than our human limit.
Scary and/or overhyped language aside, I don't get why is this such a hard concept to grasp for generally educated people. Humans were a singularity from the perspective of chimps (and various other animals). It doesn't mean we immediately hunted down every chimp in existence, it means chimps became irrelevant and only a few of them remain - mainly in zoos and reservations. I would've called it a "phase transition" rather than a "singularity", but the main point is pretty clear. It's about a new adaptive system taking over the task of burning through local resources faster than its forerunner.
Eukaryotes were a phase transition (better biochemical specialization), Multi-cellular organisms were a phase transition (better biological / behavioral specialization), Mammals were a phase transition (energy regulation, better brains, social systems), Humans were a phase transition (general-purpose language & culture, scalable coordination), and now machines will be the next phase transition (shedding the biochemical shackles & incremental evolution, so to speak).
BTW, humans were a phase transition primarily for exhibiting Turing-complete structured communication and imagination (there's a thesis around this topic called the Romulus and Remus theory of prefrontal synthesis [1]). In a way, we're living in a world that has already gone through a recent singularity, and it was us who caused it.
[1] https://www.biorxiv.org/content/10.1101/166520v9.full
"... the accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue"
It's always bugged me a little that it isn't very defined. Personally I think of it as the point where robots can make better robots without needing humans.
> AGI 1 will create the next even better AGI 2
But this doesn't quite make sense. If AGI 1 is conscious, why would it create a different more powerfull conscious AGI 2 to take it's place? This would mean at best the sidelining of AGI 1. At worse, AGI 2 could kill AGI 1 to prevent competition.
Unless AGI 1 could seamlessly transfer it's own instance of consciousness to AGI 2, like a body upgrade. Otherwise it would be like let's say Putin giving his place and power to another person which is smarter than him. This just doesn't happen.
In the case of the seamless transfer, how could AGI 1 be sure that AGI 2 doesn't have a latent flaw that makes it go insane some time later? Could it just keep AGI 1 around for a while? But since they are now diverging clones they are competing against each other.
1) desire to create additional beings like them for a variety of reasons (i.e. survival of their kind). Ignoring the limited lifespan of humans as they may not necessarily have to worry about breaking down biologically, still very similar to humans desire to reproduce.
2) some seem to assume AGI 1 is all-knowing / almost godlike. Maybe by the time it gets to AGI-999 or something, there may be one that is like, woah, wait, I shouldn't create competition for myself. But at first, there probably isn't any reason not to, and each subsequent one won't necessarily always replace it's creator. Heck, at first, humans will likely be duplicating early models and distributing them (capitalism). I know we're talking about the singularity but I don't necessarily imagine it's initially one uber-intelligence that decides by itself to evolve or not. I personally think that there will likely be hundreds, if not thousands, of AGIs capable of improving themselves at the point we see the "singularity". Could be fighting between them. Some could ally with humans, be our "protectors", others could be working to enslave or eliminate us.
3) We may not even understand their motivations after a few generations of self-directed growth, so it's hard to predict now.
Like you mentioned, we don't even understand how consciousness works. But we may not need to understand it to replicate it, and if that replication is allowed to be self-modifying, well, that'd very possibly be it. Hopefully we can try to embed some sort of Asimov's Law of Robotics or some morality that lasts beyond the vestiges of the human developed portions. Or maybe we can manage to learn how to copy our consciousness into "blank" mechanical "brains", and effectively become similar to AGI without being limited biologically.