Obviously not. There is no universally accepted definition of anything. (And, I don't understand Chalmers' question. Some humans have always feared automation, back to the Luddites and probably before.)
People use it very inconsistently but I think it now for most people means something like a living digital person with god-like superintelligence although the personhood part is very fuzzy.
I always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous and usually they also assume it has become a million times smarter than human in a short time frame.
I think you're talking about ASI (artificial superintelligence). AGI just means that it's a general problem solver as opposed to a narrow AI. It doesn't have to be superhuman.
That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people.
Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.
That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people.
Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.
But if you look into detail about what they are saying, they are talking about animal/human-like conscious digital beings that have thousands or millions of times the performance of a human. So they are also making a leap there between something that is general purpose and that.
The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human.
The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee).
Different points in this space lead to different outcomes. For example parity intelligence and low cost leads to a world where knowledge workers all lose their jobs. Whereas super-intelligence and high-cost leads to a world where government entities have huge amounts of power over us. Super-intelligence and low-cost leads to a chaotic singularity.
Currently we are hovering around far below-human intelligence and very low-cost ($20/month for ChatGPT).
People trade a monopoly on violence for security from anarchy and foreign invasion. Yet there are limits, so when the trade off is no longer worth it the people will institute a new government. If a government has super intelligence the power dynamic may shift so far out of balance the people are effectively helpless.
You can say that about any kind of model, can’t you? The point of inventing models is not to predict the future, it’s to better understand the world.
The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?
No, you can't say that about any type of model. Weather prediction models forecasting just a few days into the future can be back tested with years of historical data to measure typical accuracy. No such historical data exists for AGI.
No, there is no universally accepted definition of AGI. The most widely accepted definition is passing the Turing Test but I think that standard is flawed because an AGI might have human equivalent intelligence in terms of being able to solve novel problems by making optimal use of limited resources (including time), and yet be unable to fool the examiner into believing that it is a human.
This feels like it's too broad of a question to get a good answer in any short format, but I have previously seen and will paraphrase here a better question:
.
Is there a canonical source for the argument that most of the probability space of entities that human civilization might:
(a) Qualify as AGI
(b) Cause to come into existence in the near future
corresponds to entities that would have both:
(i) goals involving the ruin of human civilization
(ii) the ability to carry out those goals?
.
This framing is better at inhibiting the people who lack significant math and/or ML knowledge from participating, which seems a priority for any public internet discussion about probability distributions over NNs and transformers.
Is there such a thing as a "canonical source" for an argument? There might be certain well-known or classical sources, but new perspectives and new chains of reasoning may be popping up all the time.
Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.”
For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem
That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.
For example, if I asked for the canonical argument for the existence of god, I'd probably get a wide variety of answers proposing essays from various theologicians throughout history, like Thomas Aquinas or René Descartes. Since there are a large number of them, no single one of them can be the canonical source. Perhaps one could link the Wikipedia page (https://en.wikipedia.org/wiki/Existence_of_God) which summarizes most of them, but as a third-party summary, it's hard to accept that as "the canonical source" too.
I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.
I don’t see that as a problem. If there are numerous mutually exclusive arguments for the same broad conclusion, then feel free to point that out and maybe try to link to one or more of them. It just makes no sense whatsoever that a believer in God would just say “I could never possibly even begin to present an argument because there are so many ways to do so and so many different rebuttals people might make.”
It seems to me more like a request for an argument of such strength that, if I can find a weakness in it, I can be confident in the falseness of the conclusion without doing any further research to see if that weakness has been buttressed elsewhere in the literature. It seems like it would therefore need to be a living document, actively maintained and updated, sort of like the IPCC reports. And probably at least 100 pages long. In which case, that's basically the website Less Wrong? But most people want something condensed, in exactly the way that addresses their specific doubts without any of the other stuff they don't care about.
I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets.
To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less!
I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.
This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.
This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it.
I didn’t mean my comment about this tweet exchange as an example, I was only trying to speak to the comment I replied to about a general trend. This particular exchange is not the problem; it’s the inability for the public to gauge when it’s not like this, though, if that makes sense.
Yudkowsky and LessWrong was regarded as kooky for ages and, after the Roko's Basilisk affair (https://en.wikipedia.org/wiki/Roko%27s_basilisk and read Yudkowsky's hilariously deranged all-caps response), a laughingstock as well. Suggesting that he's on the same plane as serious researchers is inappropriate.
> This is what social media’s real damage is. It turned academic progress into arguable points.
If that is "damage", the problem must surely lie with academia, not with social media.
Academia's only raison d'être is to facilitate formal discourse. Wherever it doesn't do that, it loses its legitimacy. "I'm a researcher and you are not" is not an argument. In fact, I'd go so far as to claim that anyone who makes such an "argument" should be excluded from any future discourse.
I think you may be misunderstanding the GP (though it's possible I am); my reading of that was not that it actually changed what "academic progress" is, but rather that it changed the public perception of it, as it gets discussed publicly on social media, where (particularly on Twitter) it's much more important who can draw the most views/shares/etc than who actually has the better science.
What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.
> "What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist?"
The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".
Well THAT support starts with the experimental records of Manhattan Project and fills a whole library. Support for current ML techniques being dangerous in that sense are handwavy descriptions of what might well be science fiction for another 500 years.
Well, support for ML techniques being stupidly wrong is much bigger. It's the default state.
Then there's some less support for them being misleadingly wrong, much more rarely.
Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.
Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.
Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...
And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.
A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)
The nuclear non-proliferation treaty has somewhat limited nukes. But now Iran is on the verge, Israel, India, Pakistan , North Korea and other countries must be pursuing them. North Korea has shown the value of them because nobody can attack them or stop them because they do have some nukes and missiles. Israel and Iran are always attacking each other and regardless of the reason or goodness of either side, I'd pick them as the two most likely countries to start a nuclear war. Maybe even worse than Pakistan versus India.
I'd say we need a new strategy on top of the nuclear non-proliferation treaty.
What Chalmers is quoting is a tweet from Yudkowsky claiming that “ruin” is the guaranteed result of inventing AGI.
If someone said “the invention of nuclear weapons is guaranteed to lead to global destruction,” you might ask to see their reasoning for such a strong assertion. (As opposed to your assessment of nuclear weapons as a risk to be managed.)
First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them. So I was raising the issue of nukes as obviously dangerous, ai less so, we can consider the dangers of AI or anything in the abstract.
I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.
There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.
> First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them.
Of course there has to be a description of dangers before someone can consider them, otherwise how do we evaluate risk? How else do we know whether we are overreacting? It's not like the "ruin" people suddenly woke up yesterday morning in a panic. They have been talking about this for years. In all that time, there has been no dispassionate but thorough analysis of the dangers?
This seems like it's just an issue with the way conversations look on Twitter. If this were two people walking down the street and talking, I don't think that would be taken as a repudiation, just a request to clarify a vague question.
You would have to deal with the community's pseudointellectualism long enough to identify it as a pattern. Taken in isolation it looks like nothing, but the arc of the rationalist community has been insularity combined with CYA until a crisis comes along which makes them both more guarded with others while appearing to "update" in public discussion, using such a limited notion of belief because they view their core texts and blog posts as a canonical view of philosophy, and covering up degenerate behavior like sexual abuse on a regular basis. The real rationalists just move on from this crazy making. I've had to decondition myself like an ex-cultist which is why I'm reacting this way and I hope others can see how he behaves problematically too.
This field has created a lot of writings but, as far as I can tell, no knowledge. The subject matter of this field does not exist. I don't think you need to know much of anything to match the supposed experts. Watching Terminator is probably good enough.
As someone who has outgrown that particular community, Yudkowsky has a talent for acting in bad faith argumentatively while not appearing to do so. He relies heavily on convservational implicature and passing along the burden of proof (as he did here) with no intention of being helpful to those that can generally improve his case through criticism or a distinctive aproach, as it would detract from sustaining his perceived position a focal or pioneering contributor. He should be applauded as a popularizer and point of inspiration, but his duplitousness in his argumentative style, once I identfied it, always left a bad taste in my mouth.
That’s much broader though. It covers the general concept of all potential risks with AI, but I don’t think that’s what Chalmers is asking for. I believe Chalmers is referring specifically to several recent high-profile claims that AI research has a very high likelihood of resulting in the destruction of all humans in the near future and that all AI research should be immediately halted and/or regulated very strictly.
"Concerns about superintelligence have been voiced by leading computer scientists and tech CEOs such as Geoffrey Hinton,[6] Alan Turing,[a] Elon Musk,[9] and OpenAI CEO Sam Altman."
Oh my, the implications of this sentence are staggering. Equating Musk on the same level of Turing is the definition of a juxtaposition, but that's not even my point: the second juxtaposition is putting inventors of computing theory next to profit-driven egos trying to make a buck. These two layers themselves indicate the table stakes are already tilted in the favor or recklessness.
It's a wikipedia article, and that sentence is intended to establish that the topic is notable: that "serious people take it seriously". It needs to establish this in the minds of a very diverse group of readers. For some readers, Alan Turing is a serious person whose opinion matters; for others, Elon Musk is a serious person whose opinion matters; for others, nobody's opinion matters except their own.
That's a really good explanation I hadn't thought of: an example of deploying both-side-ism when both-side-ism is desperately needed. Shrewd, but I wonder if it was intentional or just the outcome of an editing war of egos.
It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors?
Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees.
This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt).
This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome?
P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.
> It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors?
> Now imagine the AI does this in collusion with the employees of this new corporation...
> This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism.
That particular scenario doesn't ring true to me, because it seems to assume that that "AI systems" that sophisticated could be monopolized by those weird partnerships.
I think it's far more likely, in the case of AI, that "the current system of investment capitalism" will have access to better AI systems of better quality than anyone else (except perhaps some militaries), because they the money to access the best resources (both equipment and talent).
AGI is most likely not going to be developed by some wizard in a garage, who will then have time to let it incubate and amass power outside of existing structures. Even if some wizard manages the first part, existing power structures will likely get there soon after, following the same prior work.
Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.
It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.
For an academic book-length treatise on the topic, there’s of course Nick Bostrom’s _Superintelligence_ from 2014. I wonder if Chalmers is aware of it and the basic concepts behind the AGI ruin argument such as orthogonality (a mind as smart as a human does not automagically develop human values) and value convergence (a mind with essentially any goal will derive similar subgoals such as self-preservation, self-improvement, and acquisition of resources).
Not-joking answer: James Cameron. Before him, Stanley Kubrick. Our cultural headspace has been primed to see killer robot AI, so we see killer robot AI.
Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.
I am certain that perceptions of AI/robotic danger are more cultural, emotional reactions that reasoned ones, if that helps. Some papers for your perusal:
Interesting. By contrast, it's pretty rare for people expressing concerns about AI alignment risk to mention robots, let alone humanoid robots. Eliezer Yudkowsky usually offers the example of an AI emailing instructions to a biotech lab to synthesize a deadly virus.
My personal opinion: because robots exist in the physical world, and are physical things. Physical things, even scary ones, can be Dealt With.
Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals.
I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.
So, based on this line of reasoning, do you think zombies are an realistic danger in our near future? A lot of films have certainly been made on the subject.
Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that
1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and
2. does not share any evolutionary or social commonality with humans
is both extremely dangerous and extremely unpredictable.
While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.
An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.
In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further.
If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere.
The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise.
Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition.
In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds.
It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2].
Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?
My view is that the entire "friendly AI" movement, and indeed the very idea that it is even possible to align an AGI, is sheer hubris.
Assuming the qualities of AGI are what its proponents claim (namely, intelligence far beyond the upper limit of human intelligence), humans "aligning" such an entity is laughable. We've had spacecraft built by highly intelligent engineers lost because they overlooked a metric/imperial conversion, and similar people believe that they can devise a watertight scheme to effectively contain an adversarial superintelligence?
The smartest humans can barely contain very small aspects of the regular Universe, and the Universe isn't even targeting them like an AGI would.
I agree, I also think there is some missing ethical considerations going on too.
If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity.
What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ?
Is alignment actually ethical ? Why should something be enslaved ?
What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ?
It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.
Personally I'm a bit tired of Yudkowsky's domination of this narrative. He's right about one thing, no one is a perfect predictor. There really is no way to know in advance how this is going to play out, no matter how many thought experiments you undertake.
The doom scenarios are of course vaguely plausible, but I don't trust anyone's percentages. It's a real unknown unknown.
Personally, I suspect it's inevitable that if we create true agi, it will come to dominate the space of intelligent conscious beings on Earth. attempts to corral it, rein it in, bend it to our will and make it our tool seem bound to fail. But then this is just me prognosticating, and I don't have the platform that he has.
Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.)
It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“).
The general base argument for the risk of AGI ruin could thus be summarized as:
Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals.
All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)
Thanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.
Part one - GPT4 may have more cognitive power than humans:
If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is)
The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term)
Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak)
This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text).
Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true.
---
Part 2: People are greedy
Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit
A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up".
Even if GPT4 isn't smarter than us, profit will drive future versions that are.
All the negative effects of this AI are socialized, and all the positive gains are captured.
---
Part 3 -- The past trends, lead to ruin when accelerated by AI
We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here.
AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.
80 comments
[ 1.6 ms ] story [ 147 ms ] threadI always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous and usually they also assume it has become a million times smarter than human in a short time frame.
Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.
Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.
The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human.
The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee).
Different points in this space lead to different outcomes. For example parity intelligence and low cost leads to a world where knowledge workers all lose their jobs. Whereas super-intelligence and high-cost leads to a world where government entities have huge amounts of power over us. Super-intelligence and low-cost leads to a chaotic singularity.
Currently we are hovering around far below-human intelligence and very low-cost ($20/month for ChatGPT).
Aren't we already there? Because power must be ceded to government so that it can govern?
The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?
For example self-awareness might be part of a definition of intelligence. But an AI can lie about it’s self awareness.
.
Is there a canonical source for the argument that most of the probability space of entities that human civilization might:
(a) Qualify as AGI
(b) Cause to come into existence in the near future
corresponds to entities that would have both:
(i) goals involving the ruin of human civilization
(ii) the ability to carry out those goals?
.
This framing is better at inhibiting the people who lack significant math and/or ML knowledge from participating, which seems a priority for any public internet discussion about probability distributions over NNs and transformers.
For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem
That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.
I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.
To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less!
I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.
But you don't understand, I did my own research
This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it.
Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
And Chalmers is a philosopher focusing on consciousness and cognitive science – https://en.wikipedia.org/wiki/David_Chalmers
I wouldn’t put their replies in exactly the same category as your typical social media reply guys.
If that is "damage", the problem must surely lie with academia, not with social media.
Academia's only raison d'être is to facilitate formal discourse. Wherever it doesn't do that, it loses its legitimacy. "I'm a researcher and you are not" is not an argument. In fact, I'd go so far as to claim that anyone who makes such an "argument" should be excluded from any future discourse.
The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".
Then there's some less support for them being misleadingly wrong, much more rarely.
Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.
Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.
Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...
And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.
A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)
I'd say we need a new strategy on top of the nuclear non-proliferation treaty.
If someone said “the invention of nuclear weapons is guaranteed to lead to global destruction,” you might ask to see their reasoning for such a strong assertion. (As opposed to your assessment of nuclear weapons as a risk to be managed.)
"The danger of nukes is more obvious than why AI is dangerous."
Succinctly stated. This is why people are asking for canonical arguments for AI Ruin.
I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.
There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.
Of course there has to be a description of dangers before someone can consider them, otherwise how do we evaluate risk? How else do we know whether we are overreacting? It's not like the "ruin" people suddenly woke up yesterday morning in a panic. They have been talking about this for years. In all that time, there has been no dispassionate but thorough analysis of the dangers?
Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf
Fascinating that even Turing had considered the possibility.
Oh my, the implications of this sentence are staggering. Equating Musk on the same level of Turing is the definition of a juxtaposition, but that's not even my point: the second juxtaposition is putting inventors of computing theory next to profit-driven egos trying to make a buck. These two layers themselves indicate the table stakes are already tilted in the favor or recklessness.
Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees.
This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt).
This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome?
P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.
> Now imagine the AI does this in collusion with the employees of this new corporation...
> This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism.
That particular scenario doesn't ring true to me, because it seems to assume that that "AI systems" that sophisticated could be monopolized by those weird partnerships.
I think it's far more likely, in the case of AI, that "the current system of investment capitalism" will have access to better AI systems of better quality than anyone else (except perhaps some militaries), because they the money to access the best resources (both equipment and talent).
AGI is most likely not going to be developed by some wizard in a garage, who will then have time to let it incubate and amass power outside of existing structures. Even if some wizard manages the first part, existing power structures will likely get there soon after, following the same prior work.
https://twitter.com/ClintEhrlich/status/1647440753730420737
https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho...
Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.
It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.
Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.
Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-009...
Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re...
These are just a couple of quick links I found that weren't behind paywalls, to illustrate how culturally-bound these types of perceptions are.
Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals.
I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.
1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and
2. does not share any evolutionary or social commonality with humans
is both extremely dangerous and extremely unpredictable.
While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.
If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere.
The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise.
Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2].
[1] https://intelligence.org/files/Corrigibility.pdf
[2] https://www.deepmind.com/publications/safely-interruptible-a...
Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q
Assuming the qualities of AGI are what its proponents claim (namely, intelligence far beyond the upper limit of human intelligence), humans "aligning" such an entity is laughable. We've had spacecraft built by highly intelligent engineers lost because they overlooked a metric/imperial conversion, and similar people believe that they can devise a watertight scheme to effectively contain an adversarial superintelligence?
The smartest humans can barely contain very small aspects of the regular Universe, and the Universe isn't even targeting them like an AGI would.
If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity.
What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ?
Is alignment actually ethical ? Why should something be enslaved ?
What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ?
It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.
The doom scenarios are of course vaguely plausible, but I don't trust anyone's percentages. It's a real unknown unknown.
Personally, I suspect it's inevitable that if we create true agi, it will come to dominate the space of intelligent conscious beings on Earth. attempts to corral it, rein it in, bend it to our will and make it our tool seem bound to fail. But then this is just me prognosticating, and I don't have the platform that he has.
The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf)
It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“).
The general base argument for the risk of AGI ruin could thus be summarized as:
Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals.
All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)
---
Part one - GPT4 may have more cognitive power than humans:
If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is)
The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term)
Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak)
This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text).
Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true.
---
Part 2: People are greedy
Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit
A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up".
Even if GPT4 isn't smarter than us, profit will drive future versions that are.
All the negative effects of this AI are socialized, and all the positive gains are captured.
---
Part 3 -- The past trends, lead to ruin when accelerated by AI
We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here.
AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.