Nothing about this risk or the statement implies AGI is real, because the risk exists in wide scale use of existing technology. It's the risk of belief in algorithmically derived information, and deployment of autonomous, unsupervised systems.
It's great they signed the statement. It's important.
I don't get this argument at all. Why does the fact that you doubt the intentions one of the signatories mean we can disregard the statement? There are plenty of signatories (including 3 turing award winners) who have no such bias.
Yeah, fair enough, it doesn't necessarily invalidate the statement. But it's odd, don't you think? It's like if a group released a public statement that said "Stop Oil Now!" and one of the signatories was Exxon-Mobil. Why would you let Exxon-Mobil sign your statement if you wanted to be taken seriously?
What would Max Tegmark, Geoffrey Hinton, Yohshua Bengio (to name a few) have absolutely anything to do with FAANG ?
They're completely independent AI researchers and geniuses spending their own free time on trying to warn you and others of the dangers of the technology they've created to help keep the world safer.
Seems like you're taking a far too overly cynical position ?
It is coming before climate change. No matter which group "put it" reality doesn't care. Humanity will not get extinct in the next 10 years by climate but many AI scientists think there is a chance this happens with AI.
The US spent trillions of 2020 dollars trying to limit the threat of nuclear war, and this statement says that AI risk should be seen as a similar level of threat.
It seems to be a PR related statement. For example, OpenAI's Sam Altman has signed it but is as far as I can understand very resistant to actual measures to deal with possible risks.
I don't think that's a fair assessment. He favors government oversight and licensing. Arguably, that would entrench companies with deep pockets, but it's also a totally reasonable idea.
> "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back," he told Reuters. "They are still talking about it."
And really, what we'll see is the current EU AI Act as-is is probably not strong enough and we'll almost certainly see the need for more in the future.
Right now, he's openly calling for regulation. That's a verifiable fact.
It's very possible that when specific proposals are on the table, that we'll see Altman become uncooperative with respect to things that don't fit into his self-interest. But until that happens, you're just speculating.
Boiling it down to a single sentence reduces ambiguity. Also, given that AI x-risk analysis is essentially pre-paradigmatic, many of the signatories probably disagree about the details.
As the pre-amble to the statement says: they kept the statement limited and succinct as there may be disagreement between the signatories about the exact nature of the risk and what to do about it.
Signatures from massive tech giants that on one hand are saying "hold on this is scary, we should slow down" but also "not us, we're doing fine. You should all slow down instead" mean that this is a bit of a empty platitude.
I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.
It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.
Is more research really going to offer any true solutions? I’d be genuinely interested in hearing about what research could potentially offer (the development of tools to counter AI disinformation? A deeper understanding of how LLMs work?), but it seems to me that the only “real” solution is ultimately political. The issue is that it would require elements of authoritarianism and censorship.
A lot of research about avoiding extinction by AI is about alignment. LLMs are pretty harmless in that they (currently) don't have any goals, they just produce text. But at some point we will succeed in turning them into "thinking" agents that try to achieve a goal. Similar to a chess AI, but interacting with the real world instead. One of the big problems with that is that we don't have a good way to make sure the goals of the AI match what we want it to do. Even if the whole "human governance" political problem were solved, we still couldn't reliably control any AI. Solving that is a whole research field. Building better ways to understand the inner workings of neural networks is definitely one avenue
I see. Thanks for the reply. But I wonder if that’s not a bit too optimistic and not concrete enough. Alignment won’t solve the world’s woes, just like “enlightenment” (a word which sounds a lot like alignment and which is similarly undefinable) does not magically rectify the realities of the world. Why should bad actors care about alignment?
Another example is climate change. We have a lot of good ideas which, combined, would stop us from killing millions of people across the world. We have the research - is more “research” really the key?
Intelligence cannot be 'solved', I would go on to further say that an intelligence without the option of violence isn't an intelligence at all.
If you suddenly wanted to kill people, for example, then could probably kill a few before you were stopped. That is typically the limits of an individuals power. Now, if you were a corporation with money, depending on the strategy you used you could likely kill anywhere from hundreds to hundreds of thousands. Kick it up to government level, and well, the term "just a statistic" exists for a reason.
We tend to have laws around these behaviors, but they are typically punitive. The law realizes that humans, and human systems will unalign themselves from "moral" behavior (whatever that may be considered at the time). When the lawgiver itself becomes unaligned, well, things tend to get bad. Human alignment typically consists of benefits (I give you nice things/money/power) or violence.
There is a way, in my opinion: distribute AI widely and give it a diversity of values, so that any one AI attempting takeover (or being misused) is opposed by the others. This is best achieved by having both open source and a competitive market of many companies with their own proprietary models.
Personalization, customization, etc.: by aligning AI systems to many users, we benefit from the already-existing diversity of values among different people. This could be achieved via open source or proprietary means; the important thing is that the system works for the user and not for whichever company made it.
It's difficult as most of the risk can be reinterpreted as a highly advanced user.
But that is where some form of hard personhood zero proof mechanism NEEDS to come in. This can then be used in conjunction with a Ledger used to track deployment of high spec models. And create an easy means to Audit and deploy new advanced tests to ensure safety.
Really what everyone also need to keep in mind at the larger scale is that final turing test with no room for deniability. And remember all those Sci-fi movies and how that Moment is portrayed traditionally.
> It’s hard to take these safety concerns seriously
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them.
And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power.
Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine.
> non-proliferation treaties were a scam
And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time.
The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.
> There are no real consequences if you are backing off, and you can do it in any time.
That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
I think it really depends.
E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries.
India or Pakistan are not part of the treaties and for some reason, we don't see big problems.
There is only China left who might leave the treaty in the first place, anymore.
And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means?
Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.
Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body.
The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.
Germany, for example would disagree with you. They believe violent speech is an act of violence in itself.
>AI is an application of math.
It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.
And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.
Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
> It turns out that people hook computers to 'things' that exist in the physical world.
But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing.
However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
Those tangible elements would conceivably become the danger, not the AI using those elements. Again, the "guns don't kill people, people kill people" take is all well and good, but well outside of this discussion.
Maleficent humans are constantly trying to use these elements for their own gain, often with little to no regards to other humans (especially out groups). This happens both individually, in small groups, in large organizations and even multiple organization colluding. Both criminal, terrorist, groups at war, along with legal organizations such as exploitative companies and regressive interest organizations, et.c.. And we have tools and mechanisms in place to keep the level of abuse at bay. Why and how are these mechanisms unsuitable for protecting against AI?
>Why and how are these mechanisms unsuitable for protecting against AI?
The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.
> Both criminal, terrorist, groups at war
Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.
Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?
Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.
False premise. One can start new threads about complimentary subjects and they can be thought about in parallel. You don't have to try and shove all of the worlds concepts into just one thought train to be able to reason about them. That's how you make spaghetti.
>It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking.
"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.
The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").
So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.
I don't understand this argument (the "Terminator" scenario). AI could conceivably self replicate and evolve as software but it still needs hardware to run on, power, networking etc etc. There's no better way to kill that than to start a nuclear war or engineer some super virus that kills everyone.
It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).
Sure. I recommend reading superintelligence by Nick Bostrom.
But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.
Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.
The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.
Superintelligence and Life 3.0 seem to come up as recurring references in the discussion. I've only read synopses of both but frankly I find the argument that melevolent AI "escape" could occur without being noticed and thwarted a bit far fetched.
There's a good counterargument here[1] that seems reasonable:
"One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"
Ok - fair comment. So I found a copy of Superintelligence. The argument is pulp sci-fi at best:
"The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a “strike” in which the AI eliminates the human species and any automatic systems humans have created that
could offer intelligent opposition to the execution of the AI’s plans. This could be achieved through the activation of some advanced weapons system that the AI has
perfected using its technology research superpower and covertly deployed in the covert preparation phase. If the weapon uses self-replicating biotechnology or nano-technology, the initial stockpile needed for global coverage could be microscopic: a single replicating entity would be enough to start the process. In order to ensure a sudden and uniform effect, the initial stock of the replicator might have been deployed or allowed to diffuse worldwide at an extremely low, undetectable con-centration. At a pre-set time, nanofactories producing nerve gas or target- seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe (although more effective ways of killing could probably be devised by a machine with the technology research superpower)."
Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.
Exactly. While there is an argument to be made that people are the real danger, that is beyond the discussion taking place. It has already been accepted, for the sake of discussion, that the nuclear missile is the danger, not the math which developed the missile, nor the people who thought it was a good idea to use a missile. Applying AI to the missile still means the missile is the danger. Any use of AI in the scope of that missile is just an implementation detail.
You said that "AI is an application of math. It is not clear how that can be used directly to harm a body." I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
If the point you are trying to make is that an AI which secretly creates and deploys nuclear, biological, or chemical weapons in order to destroy all of humanity, is not an "AI risk" because it's the weapons that do the actual harm, then... I really don't know what to say to that. Sure, I guess? Would you also say that drunk drivers are not dangerous, because the danger is the cars that they drive colliding into people's bodies, and the drunk driver is just an implementation detail?
> I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
For the sake of discussion, it was established even before I arrived that those developed things are the danger, not that which creates/uses the things which are dangerous. What is to be gained by ignoring all of that context?
> I really don't know what to say to that. Sure, I guess?
Nothing, perhaps? It is not exactly something that is worthy of much discussion. If you are desperate for a fake internet battle, perhaps you can fight with earlier commenters about whether it is nuclear missiles that are dangerous or if it is the people who have created/have nuclear missiles are dangerous? But I have no interest. I cannot think of anything more boring.
I'm specifically worried that an AGI will conceal some instrumental goal of wiping out humans, while posing as helpful. It will helpfully earn a lot of money for a lot of people, by performing services and directing investments, and with its track record, will gain the ability to direct investments for itself. It then plows a billion dollars into constructing a profitable chemicals factory somewhere where rules are lax, and nobody looks too closely into what else that factory produces, since the AI engineers have signed off on it. And then once it's amassed a critical stockpile of specific dangerous chemicals, it releases them into the atmosphere and wipes out humanity / agriculture / etc.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.
Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.
To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.
Yes, but usually when people express concerns about the danger of nuclear missiles, they are only thinking of those nuclear missiles that are at the direction of nation-states or perhaps very resourceful terrorists. And their solutions will usually be directed in that direction, like arms control treaties. They aren't really including "and maybe a rogue AI will secretly build nuclear weapons on the moon and then launch them at us" in the conversation about the danger of nukes and the importance of international treaties, even though the nukes are doing the actual damage in that scenario. Most people would categorize that as sounding more like an AI-risk scenario.
Please read Life 3.0 or superintelligence. There are people that spent decades thinking about how this would happen. You spent a little bit of time and conclude it can't.
It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO.
The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
>It's not clear at all that we have an avenue to super intelligence.
All problems in reality are probability problems.
If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.
If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.
It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".
>We simply don't know how to build a system like that.
You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.
I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.
There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.
Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).
>Do you think we could wipe out all furry mammals, for example?
lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.
> It assumes that intelligence basically can overcome any obstacles
In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
> It's not clear at all that we have an avenue to super intelligence
AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.
But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.
Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.
> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.
We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?
Even if someone will, I don't think it's an "existential risk". So, yes, I'm willing to make the bet. I'm also willing to make the bet that Santa never delivers nuclear warheads instead of presents. It's why I don't cap my chimney every Christmas Eve.
Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.
> We simply don't know how to build a system like that.
Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.
"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.
We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries.
The problem is not science fiction god-mode digital quetta-smart hypercomputing.
This is about political, social, and economic influence, and who controls it.
That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI.
It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well.
[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.
And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all".
Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet.
The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote.
- Point one: If the sentient AI can launch nukes, so can your neighbor.
- Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure.
- Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.)
- Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing.
- Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it?
- Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking.
Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise.
But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.
You're a priori writing off my comment as fruitless because of your emotions and not because you actually have given it deep thought and carefully reached the conclusion that social feedback is somehow bad.
Also, the notion that "people's work" is inherently worthy of respect is just nonsensical. I do shoddy work all the time. Hell, you just casually dismissed my internet comment work as shallow and told me not to do it. Please don't post a shallow dismissal of my work.
Don't you think that this is all a bit anti-intellectual?
A counter point here is you're ignoring all the boring we all die scenarios that are completely possible but too boring to make a movie about.
The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.
You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics.
The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity.
Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.
I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric.
That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea.
There is risk in AGI. There is risk in everything. How many people are killed by furniture each year?
I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed.
I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!
> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
> The problem is, lots of "obvious" things have turned out to be very wrong.
A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.
I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.
> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
Maybe your appraisal of what counts as evidence is defective?
For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?
> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.
> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans
That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.
> the claim that it seems far too obvious that we should
They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though).
> If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people.
So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it.
> It doesn't though, does it?
It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.
2/3 deep learning Turing Price winners (Hinton and Benigo) are sufficiently shell-shocked by the rate of progress to be thrown into existential doubts (Hinton is very explicit about the fact that progress is much faster than he thought just a few years ago, Benigo speaks of how an "unexpected acceleration" in AI systems has radically shifted his perspective). Plenty of knowledgable people in the field who were not previously AI doomers are starting to sound a lot more concerned very recently.
As to the "oh it's just linear scaling of out-of-sight tech" line, well of course that itself was suprising. Gwern pushed the scaling hypothesis earlier than many and from what I remember even got pretty nasty attacks from AI insiders from it. Here's what he wrote 3 years ago: "To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.".
So sure there's some subjectivity involved here, but I'd like to see your propose some reasonable operationalization of "surprise at progress" that didn't class most laymen and insiders as suprised.
>> It doesn't though, does it?
> It does.
We seem to be miscommunicating, what I was trying to express is that my argument does not really require any appeal to authority. Trusting your lying eyes (to evaluate the progress of stuff like midjourney) and judging the quality of arguments should be enough (I spelt some reasons out here https://news.ycombinator.com/item?id=36130482, but I think hackinthebochs makes the point better here: https://news.ycombinator.com/item?id=36129980).
In fact I would still be pretty concerned even if most top AI guys were like LeCun and thought there is no real risk.
I will not deny, of course, that the fact that well known reasearchers like Hinton and Benigo are suddenly much more alarmed than they previously were and the ones like LeCun who are not seem to mostly make exceptionally terrible arguments doesn't exactly make me more optimistic.
I agree these statements from these long term researchers about them being surprised by the rate of progress are surprising.
To clarify my own thinking here, it's totally reasonable to me that people are surprised if:
1. They weren't previously aware of AI research (surely 99% of the population?)
2. They were but had stopped paying attention because it was just a long series of announcements about cool tech demos nobody outside big corps could play with.
3. They were paying attention but thought scaling wouldn't continue to work.
My problem is that people like Sam Altman clearly aren't in any of those categories and Hinton shouldn't have been in any, although maybe he fell into (3). I personally was in (2). I wasn't hugely surprised that ChatGPT could exist because I'd seen GPT-2, GPT-1, I'd seen surprising AI demos at Google years earlier and so on. The direction things were going in was kinda clear. I was a bit surprised by its quality, but that's because I wasn't really paying close attention as new results were published and the last InstructGPT step makes such a big difference to how the tech is perceived. Actual knowledge doesn't change much but once it's housetrained, suddenly it's so much easier to interact with and use that it makes a step change in how accessible the tech is and how it's perceived.
I think I was more surprised by the joining of LLMs with generators and how well AI art works. It does feel like that happened fast. But, maybe I just wasn't paying attention again.
So I guess where we differ is that I don't take their surprise at face value. The direction was too clear, the gap between the threats they talk about in the abstract and the concrete proposals are too large and too lacking in obvious logical connection; it feels like motivated reasoning to me. I'm not entirely sure what's really going on and perhaps they are genuine in their concerns but if so it's hard to understand why they struggle so much to make a convincing case given they are certainly intellectually equipped to do so.
The two posts you linked are interesting and much better argued than the website this thread is about, so I'll reply to them directly.
It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.
But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.
It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.
I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.
Do you have any better resources that you could share?
The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test.
From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.
Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.
If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
OK, which of the following propositions do you disagree with?
1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.
2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.
3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.
4. AIs that are much smarter than any human pose even graver dangers.
5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.
6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.
I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.
For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?
I think between point 3 and 4 there is a leap to talking about “danger”. Perhaps the disagreement is about what one calls “danger”. I had perhaps mistakenly assumed we were talking about an extinction risk. I’ll grant you concerns about scaling up things like surveillance but there is a leap to being an existential risk that I’m still not following.
AI will not have the instinctual drives for domination or hunger that humans do.
It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.
The only reasonable defense against rogue AI is prosocial AI.
Reading the lesswrong link, the parts I get hung up on are that it appears in these doomsday scenarios humans lose all agency. Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.
> Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.
Are you?
Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?
You could place a bunch of orders into services that syntetize DNA or proteins rn.
Some people are even working on stuff like automating protein design whith AI.
There's no reason why humans should notice anything word about a particular order on a service like that.
> 3. ... before too long we will ... see AIs that outperform humans on most tasks.
This is ambiguous. Do you mean
A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or
B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?
I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.
4. AIs that are much smarter than any human pose even graver dangers.
Sure. Why should we believe they will ever exist though?
Replying here, crossing over from the other thread.
Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".
Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.
Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.
But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"
Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?
Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans.
Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become.
The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.
> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.
Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.
General research into AI alignment does not require that those models are controlled by few corporations. On the contrary, the research would be easier with freely available very capable models.
This is only helpful in that a superintelligence well aligned to make Sam Altman money is preferable to a superintelligence badly aligned that ends up killing humanity.
It is fully possible that a well aligned (with its creators) superintelligence is still a net negative for humanity.
If you consider a broader picture, unleashing a paperclip-style cripple AI (aligned to rising $MEGACORP profit) on the Local Group is almost definitely worse for all Local Group inhabitants than annihilating ourselves and not doing that.
While I'm not on this "who's-who" panel of experts, I call bullshit.
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
tl;dr: significant near term AI risk is real and comes from the capacity for imagined ideas, good and evil, to be autonomously executed on by agentic AI, not emergent superintelligent aliens. To de-risk this, we need to align AI quickly, which requires producing new knowledge. To accelerate the production of this knowledge, the government should abandon decelerationist policies and incentivize incremental alignment R&D by AI companies. And, critically, a new public/private research institution should be formed that grants privileged, fully funded investigators multi-year funding cycles with total scientific freedom and access to all state-of-the-art artificial intelligence systems operating under US law to maximize AI as a force multiplier in their research.
> I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.
That's an excellent point.
Most of the near-term risks with AI involve corporations and governments acquiring more power. AI provides power tools for surveillance, oppression, and deception at scale. Those are already deployed and getting better. This mostly benefits powerful organizations. This alarm about strong AI taking over is a diversion from the real near-term threat.
With AI, Big Brother can watch everything all the time. Listen to and evaluate everything you say and do. The cops and your boss already have some of that capability.
Is something watching you right now through your webcam? Is something listening to you right now through your phone? Are you sure?
Ok, so if we take AI safety / AI existential risk as real and important, there are two possibilities:
1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.
2) There is a way to make AI safe without doing this.
If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.
If 2 is true, great. But it seems less likely than 1, to me.
The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.
I just don't see how the genie is put back in the bottle. Optimizations and new techniques are coming in at a breakneck pace, allowing for models that can run on consumer hardware.
I think it could be done. Or rather, instead of putting the genie back in the bottle, we could slow it down enough that we figure out how to ask it for wishes in a way that avoids all the monkey-paw's scenarios.
Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.
Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.
I have one: Levy fines on actors judged to be attempting to extend AI capabilities beyond the current state of the art, and pay the fine to those private actors who prosecute them.
This is way better than the open letter. It's much clearer and much more concise, and, maybe most importantly, it simply raises awareness rather than advocating for any particular solution. The goal appears to have been to make a statement that's non-obvious (to society at large) yet also can achieve agreement among many AI notables. (Not every AI notable agrees though--for example, LeCun did not sign, and I expect that he disagrees.)
I don't think it simply raises awareness - it's a biased statement. Personally, I don't think the advocated event is likely to happen. It feels a bit like the current trans panic in the US: you can 'raise awareness' of trans people doing this or that imagined bad thing, and then use that panic to push your own agenda. In OpenAI's case, they seem to push for having themselves be in control of AI, which goes counter to what, for example, the EU is pushing for.
In what sense is this a 'biased statement' exactly?
If a dozen of the top climate scientists put out a statement saying that fighting climate change should be a serious priority (even if they can't agree on one easy solution) would that also be 'biased'?
That's curiously the standard crackpot line. "They doubted Einstein! They doubted Newton! Now they doubt me!" As if an incantation of famous names automatically makes the crackpot legitimate.
The signatories on this are not crackpots. Hinton is incredibly influential, and he quit his job at Google so he could "freely speak out about the risks of A.I."
But that is the point. Just because scientific community is on agreement does not guarantee that they are correct. It simply signifies that they agree on something.
Note, language shift from 'tinfoil hat' ( because tinfoil hat stopped being an appropriate insult after so many of their conspiracy theories - also a keyword - became proven ) to crackpot.
In retrospect, you can find tangible proof from way back for anything that gets accepted as true. The comparison was with how climate change was discussed in the public sphere. However prominent the fossil fuel companies' influence on public discourse was at the time, the issues were not taken seriously (and sill aren't by very many). The industry's attempts to exert influence at the time were also obviously not widely known.
Rather than looking for similarities, I find the differences between the public discussions (about AI safety / climate change) quite striking. Rather than stonewall and distract, the companies involved are being proactive and letting the discussion happen. Of course, their motivation is some combination of attampted regulatory capture, virtue signaling and genuine concern, the ratios of which I won't presume to guess. Nevertheless, this is playing out completely differently so for from e.g. tobacco, human cloning, CFCs or oil.
>Extinction risk due to AI is not a generally accepted phenomenon
Why?
You, as a species, are the pinnacle of NI, natural intelligence. And with this power that we've been given we've driven the majority of large species, and countless smaller species to extinction.
To think it outside the realms of possibility that we could develop an artificial species that is more intelligent than us is bizarre to me. It would be like saying "We cannot develop a plane that does X better than a bird, because birds are the pinnacle of natural flying evolution".
Intelligence is a meta-tool, it is the tool that drives tools. Humanity succeeded above all other species because of its tool using ability. And now many of us are hell bent on creating ever more powerful tool using intelligences. To believe there is no risk here is odd in my eyes.
Perhaps open letters like this are an important step on the path to a phenomenon becoming generally accepted. I think this is called "establishing consensus".
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
risk of extinction due to AI? people have been reading too much science fiction. I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech. for the sake of conversation let's say non-AI tech is any broadly usable consumer technology before Jan 1 of 2020.
I agree that a lot of the Skynet-type scenarios seem silly at the current level of technology, but I am worried about the intersection between LLMs, synthetic biology, and malicious or incompetent humans.
But that’s just as much or more of an argument for regulating the tools of synthetic biology.
Extinction would probably require an AI system taking human extinction on as an explicit goal and manipulating other real world systems to carry out that goal. Some mechanisms for this might include:
- Taking control of robotic systems
- Manipulating humans into actions that advance its goal
- Exploiting and manipulating other computer systems for greater leverage
- Interaction with other technologies that have global reach, such as nuclear weapons, chemicals, biological agents, or nanotechnology.
It's important to know that these things don't require AGI or AI systems to be conscious. From what I can see, we've set up all of the building blocks necessary for this scenario to play out, but we lack the regulation and understanding of the systems being built to prevent runaway AI. We're playing with fire.
To be clear, I don't think I am as concerned about literal human extinction as I am the end of civilization as we know it, which is a much lower bar than "0 humans".
I don't disagree. But I believe AI is a significant multiplier of these risks, both from a standpoint of being able to drive individual risks and also as a technology that increases the ways in which risks interact and become difficult to analyze.
>> risk of extinction due to AI? people have been reading too much science fiction.
You don't think than an intelligence who would emerge and would probably be insanely smarter than the smartest of us with all human knowledge in his memory would sit by and watch us destroy the planet? You think an emergent intelligence was trained on the vast human knowledge and history would look at our history and think: these guys are really nice! Nothing to fear from them.
This intelligence could play dumb, start manipulating people around itself and it would take over the world in a way no one would see it coming. And when it does take over the world, it's too late.
honestly if you genuinely believe this is a real concern in the 2020s then maybe we're doomed after all. I feel like I'm witnessing the birth of a religion.
> I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech.
The proposed FOOM scenarios obviously borrow from what we already know to be possible or think it would likely be possible using current tech, given an proposed insanely more intelligent agent than us.
What would be in it for a more intelligent agent to get rid of us? We are likely useful tools and, at worst, a curious zoo oddity. We have never been content when we have caused extinction. A more intelligent agent will have greater wherewithal to avoid doing the same.
'Able to play chess'-level AI is the greater concern, allowing humans to create more unavoidable tools of war. But we've been doing that for decades, perhaps even centuries.
The emergence of something significantly more intelligent than us whose goal are not perfectly aligned with ours poses a pretty clear existential risk. See, for example, the thousands of species made extinct by humans.
I really thought there would be a statement detailing what the risks are but this seems more like a soundbite to be consumed on TV. Pretty disappointing.
So far the examples I've heard are: humans will ask AI to help humans solve human issues and the AI will say humans are the issue and therefore mystically destroy us somehow. Or, AI will be inherently interested in being the primary controller of earth and so destroy all humans. Or, AI will for reasons be inherently misaligned with human values. Andrej Karpathy Said it will fire nukes on us. Elon said pen is mightier than the sword and civil war is inevitable.
Can't take this seriously as long they still keep offering commercial AI services while improving the existing models they already have. (I'm not in favor of just stopping AI development and even if people claimed to stop, they probably wouldn't.)
It's like people carrying torches warning about the risks of fire.
I agree with you that some of these people (probably Sam Altman) are likely proposing this regulation out of self interest rather than genuine concern.
But I don't think the stance is necessarily hypocritical. I know nuclear engineers who advocate for better regulation on nuclear power stations, and especially for better handling of nuclear waste.
You can believe that AI is a net positive but also that it needs to be handled with extreme care.
There's a good chance that it wouldn't, but since they're the ones (initially, at least) in control of the AI they stand the best chance of not being targeted by it.
These hypothetical AI extinction events don't have to be caused by the AI deciding to eliminate humanity for its own sake like in Terminator, they could also be driven by a human in control of a not-entirely-sentient AI.
At what point do we stop pretending that the west is capitalist and accept that it's some weird corporate-cabal-command-economy? The only thing which might stop this backroom regulatory capture is the EU since they're not in on it.
>Graeber Um…that’s a long story. But one reason seems to be that…and this is why I actually had managerial feudalism in the title, is that the system we have…alright—is essentially not capitalism as it is ordinarily described. The idea that you have a series of small competing firms is basically a fantasy. I mean you know, it’s true of restaurants or something like that. But it’s not true of these large institutions. And it’s not clear that it really could be true of those large institutions. They just don’t operate on that basis.
>Essentially, increasingly profits aren’t coming from either manufacturing or from commerce, but rather from redistribution of resources and rent; rent extraction. And when you have a rent extraction system, it much more resembles feudalism than capitalism as normally described. You want to distribute— You know, if you’re taking a large amount of money and redistributing it, well you want to soak up as much of that as possible in the course of doing so. And that seems to be the way the economy increasingly works.
The issue I take with these kind of "AI safety" organizations is that they focus on the wrong aspects of AI safety. Specifically, they run this narrative that AI will make us humans go extinct. This is not a real risk today. Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc.
But of course, "AI will humans extinct" is much sexier and collects clicks. Therefore, the real AI risks that are present today are underrepresented in mainstream media. But these people don't care about AI safety, they do whatever required to push their profile and companies.
The elite class in your country views AI as a risk to their status as elites, not an actual existential threat to humanity. They are just lying to you, as usual. That is what our current crop of globalist, free-trade, open-borders elites do.
Imagine if you had an AI companion that instantly identified pilpul in every piece of media you consumed: voice, text, whatever. It highlighted it for you. What if you had an AI companion that identified instantly when you are being lied to or emotionally manipulated?
What if this AI companion could also recommend economic and social policies that would actually improve the lives of people within your nation and not simply enrich a criminal cabal of globalist elites that treat you like cattle?
The Elite class is just as apt to consolidate power with AI and rule the entire world with it. If you have a super duper AI in your pocket looking at the data around you, then they a super super super duper duper duper AI looking at every bit of data from every corner of the world they can feed the thing giving themselves power and control you couldn't even begin to imagine.
Falling into conspiratorial thinking on a single dimension without even considering all the different factors that could change belies ignorance. Yes, AI is set up to upend the elites status, but is just as apt to upset your status of being able to afford food and a house and meaningful work.
> not simply enrich a criminal cabal of globalist elites that treat you like cattle?
There is a different problem here... And that is humankind has made tools capable of concentrating massive amounts of power well before we solved human greed. Any system you make that's powerful has to overcome greedy power seeking hyper-optimizers. If I could somehow hit a button and Thanos away the current elites, then another group of powerseekers would just claim that status. It is an inane human behavior.
A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are almost certain to occur, but also for the most part have relatively minor consequences (mostly making things that are already happening worse). "Advanced AI will make humans extinct" is much less likely but has catastrophic consequences. Focusing on the catastrophic risks isn't unreasonable, especially since society at large seem to already handle the more frequently occurring risks (the EU's AI Act addresses many of them).
And of course research into one of them benefits the other, so the categories aren't mutually exclusive.
I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues.
As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing.
Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.
This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.
At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong
This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
Existential risk is one of those problems that nearly impossible to measure in most cases.
In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.
But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.
Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite.
If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care.
Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).
So the following requirements:
1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century
2) AI acquires power/means to effect human extinction
3) AI decides against coexistence with humans
Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...
This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow.
We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important.
Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions.
Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.
Although war, disease and famine may kill many people, they are extraordinarily unlikely to cause extinction per se. The human species has so many individuals that only very special risks are capable of reducing 8 billion to zero.
For most mammalian species if they have a stable 100k individuals then they are doing pretty well. So we have 80000x more humans than what would be considered a healthy population in other species.
The risks of human extinction tend to be more long-tail events that have a planet wide effect. For example, a large impact, supervolcano eruption, or engineered bio weapon. AI certainly deserves its place on that list.
Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.
If you were to take a look at the list of signatories on safe.ai, that's basically everyone who is everyone that works on building AI, what could Emily B Bender a professor of computer linguistics possibly add to the conversation and how would she be able to talk more about the "real AI safety" than any of those people?
Edit: Sorry if it sounds arrogant, I don't mean Emily wouldn't have anything to add, but not sure how the parent can just write off basically that whole list and claim someone who isn't a leader in the field would be the "real voice"?
I think we need to be realistic and accept that people are going to pick the expert that agrees with them, even if on paper they are far less qualified.
The (very small) amount of fame she's collected has come through her work in the field, and it's a field she's been in for a while; she's hardly chasing glory.
People don’t have to be chasing fame to be warped by it. She has cultivated a following of like-minded people who provide ever more positive feedback for her ever more ideological writing.
I mean she is literally dismissing people who disagree with her based on their skin color. Can we stop for a minute to wonder about the incentives that encourage that?
(and I generally like her writing and think she has interesting things to say… but I do see a reward cycle going on)
Her field has also taken the largest hit from the success of LLMs and her research topics and her department are probably no longer prioritized by research grants. Given how many articles she's written that have criticized LLMs it's not surprising she has incentives.
LLMs are in her field; they are one of her research topics and they're definitely getting funding.
We absolutely should not be ignoring research that doesn't support popular narratives; dismissing her work because it is critical of LLMs is not reasonable.
In her field doesn't mean that's what she researches, LLMs are loosely in her field but the methods are completely different. Computational linguistics != deep learning. Deep learning does not directly use concepts from linguistics, semantics, grammars or grammar engineering, which is what Emily was researcing for the past decades.
It's the same thing as saying a number theorist and a set theorist are in the same field cause they both work in the Math field.
They are what she researches though. She has published research on them.
LLMs don't directly use concepts from linguistics but they do produce and model language/grammar; it's entirely valid to use techniques from those fields to evaluate them, which is what she does. In the same vein, though a self-driving car doesn't work the same way as a human driver does, we can measure their performance on similar tasks.
Hmm I looked into it, and looked at papers/pdfs in google scholar's advanced search with her as an author that mentioned LLMs or GPT in the past 3 years. Every single one was a criticism about how they couldn't actually understand anything (e.g. "they're only trained on form" and "at best they can only understand things in a limited well scoped fashion") and that linguistic fundamentals for NLP was more important.
She's the first author of the stochastic parrots paper, and she's fairly representative of the group of "AI safety" researchers who view the field from a statistical perspective linked to social justice issues. That's distinct from the group of "AI safety" researchers who focus on the "might destroy humanity" perspective. There are other groups too obviously -- the field seems to cluster into ideological perspectives.
Current topic aside, I feel like that stochastic parrots paper aged really poorly in its criticisms of LLMs, and reading it felt like political propaganda with its exaggerated rhetoric and its anemic amount of scientific substance e.g.
> Text generated by an LM is not grounded in communicative
intent, any model of the world, or any model of the reader’s state
of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have
the ability to do that.
I'm surprised its cited so much given how many of its claims fell flat 1.5 years later
It's extremely easy to publish in NLP right now. 20-30% acceptance rates at even the top end conferences and plenty of tricks to increase your chances. Just because someone is first author on a highly cited paper doesn't imply that they're "right"
She’s contributed to many academic papers on large language models and has a better technical understanding of how they work and their limitations than most signatories of this statement, or the previous widely hyped “AI pause” letter, which referenced one of her own papers.
I find her and Timnit Gebru’s arguments highly persuasive. In a nutshell, the capabilities of “AI” are hugely overhyped and concern about Sci-Fi doom scenarios is disingenuously being used to frame the issue in ways that benefits players like OpenAI and diverts attention away from much more real, already occurring present-day harms such as the internet being filled with increasing amounts of synthetic text spam.
Thanks for the link, I read it with interest but philosophically it is a flawed argument (IMHO). It's a nonsequitur or something, for the following two reasons.
First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.
Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here are the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.
(There's a well known chart, the hierarchy of disagreements. The most sophisticated intellectual disagreements point out what's wrong with the argument. Less sophisticated disagreements do things like point out alternatives, without pointing out what the critical mistake is. The critical mistake in this case hinges on whether the premise of AGI is true or not. That's the crux of disagreement. Substituting that with short-term issues, which are valid in themselves, are the example of a lower-level of argumentation. And of course even lower levels of argumentation are bad-faith readings and so forth, I forget but the chart had several levels. It's funny that professional academics nevertheless don't practice this and so get into endless, intellectually unsatisfactory debates.)
So I think this is actually an example of different factional experts constantly talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.
> First, I’m inclined to think that longtermism is an invalid and harmful ideology
It is, but (aside from as a sort of sociological explanation of the belief in AGI risk) that’s mostly beside the point when discussing the problems AGI x-risk. The problem of AGI x-risk is that it is an entirely abstract concern which does not concrete flow from any basis in material reality, cannot be assessed with the tools of assessing material reality, and exists as a kind of religious doctrine surrounded by rhetorical flourishes.
> The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns.
They are way of understanding why people who seem (largely because they are) intelligent and competent are trying to sell such hollow arguments as AGI x-risk. They aren’t, you are correct, a logical rebuttal to AGI risk, nor are they intended as that; the only rebuttal is the complete absence of support for the proposition. They are, however, a tool that operates outside the realm of formalized debate that addresses the natural and useful cognitive bias that itself is outside of the realm of formalized debate that says “smart, competent people don’t tend to embrace hollow positions”.
> Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, “No, AGI is a false problem and here a the real problems”.
1. If they couldn’t, it wouldn’t be a “reading comprehension issue”, and
2. They can, for the simple reason that there is no material support for the “AGI is a real problem” argument.
> Bender et. al are just sidestepping the moot point,
A point being moot in the sense that AGI x-risk is is a reason to sidestep it. (The danger of using auto-antonyms.)
> I think honest experts could simply say, ultimately we don’t really know what will happen.
To the extent that is accurate, that is exactly what the Bender/Gebru/Mitchell group does. The problem is thinking “we don’t have any information to justify any belief on that” is one sided and means that utility of AGI is somewhere between 0 and the negative infinity that the x-risk crowd calculates from (some unspecified non-zero finite probability) times (infinite cost), whereas the reality is that we have as much reason to believe that AGI is the only solution to an otherwise certain existential calamity as to suppose it will lead to one. The utility is somewhere between positive infinity and negative infinity.
A point being moot is a reason to agree to disagree, if they can't agree on the premise. But they need to say that. If I were writing a letter, I would say it because that's just being sensible.
This isn't about logical debate. This is about reasonable, non-sophistic writing at the college level or higher. And there are basic standards, like if they don't know the future then they must explicitly acknowledge that. Not rhetorically "do that" in the essay. Literally write it out in sentences. They didn't.
I can think of 3 examples where such explicitness was done. Chomsky's letter gave explicit reasons why AGI is a false issue (and he was pilloried for it). My computer science professors literally, in their deep learning class and in their theoretical machine learning research seminars, have literally acknowledge that we don't know almost anything about the fundamentals nor the future. That scientific humility and level of intellectual conscientiousness is needed. That is absent in this discourse between the experts. And note, by that, I also include the 22-word "letter" which doesn't actually explain why Hinton and the rest of the signatories think AGI is an existential risk, what their specific reasons (your "material evidence") for that are.
I have no issue with her choice of pronouns. I just find it odd that she states them when ~100% of the population would infer them correctly from her name Emily (and picture). My guess is she put them there for ideological signaling.
This is unnecessarily cynical. Why should people who are androgynous or trans be the only ones who state pronouns? By creating a norm around it we can improve their comfort at extremely minimal cost to ourselves.
I disagree, but HN is probably not the right place for this kind of debate. Also, it seems that you don't follow your own recommendation (on HN at least).
> My guess is she put them there for ideological signaling.
Yes, and you attacking her for that choice is also ideological signaling. This isn't a case where one choice is totally devoid of signal and the other is pejorative-of-the-day.
Many people believe that using normative pronouns makes those who use different pronouns more comfortable. They may be right, wrong, misguided, whatever. But it's a well-meaning and harmless act, and extremely strange to mention in the same breath as overt racism.
Reads like a caricature of the people leading these causes on AI safety. Folks that are obsessed with the current moral panic to the extent that they will never let a moment go by without injecting their ideology. These people should not be around anything resembling AI safety or "ethics".
I might look past the fact that she was sexist/racist if the topic of interest wasn't ethics. But since it is, I'd say those quoted tweets are pretty relevant.
the issue with hacker news comments these days is people don't actually do any due diligence before posting. center for ai safety is 90% about present AI risks and this ai statement is just a one off thing.
Don't characterize the public as that stupid. The current risks of AI are startling clear to a layman.
The extinction level even is more far fetched to a layman. You are the public and your viewpoint is aligned with the public. Nobody is thinking extinction level event.
Extinction is exactly what this submission is about.
Here is the full text of the statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
By "extinction", the signatories mean extinction of the human species.
You hear similar arguments from those who believe climate change is happening but disagree with current efforts to counter-act it. The logic being that right now climate change is not causing any major harm and that we can't really predict the future so there's no point in worrying about what might happen in a decade or two.
I don't think anyone is arguing that right now climate change or AI is threat to human civilisation. The point is that there are clear trends in place and that those trends are concerning.
On AI specifically, it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
In the near-future AI may help us advance biotech research and it could aid in the creation of bioweapons and other destructive capabilities.
Longer-term risks (those maybe a couple of decades out) become much greater and also much harder to predict, but they're worth thinking about and planning for today. For example, what happens when humanity becomes dependant on AI for its labour, or when AI is controlling the majority of our infrastructure?
I disagree but can understand the position that AI safety isn't humanities number one risk or priority right now, however I don't understand the dismissive attitude towards what seems like a clear existential risk when you project a decade or two out.
>it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
How are you building this progression? Is there any evidence to back up this claim?
I am having a hard time discerning this from fear-mongering.
If AI improves by 0.0001% per year on your favorite intelligence metric there will eventually be a point where it surpasses human performance and another point where it surpasses all humans combined on that metric. There is danger in that scenario.
The problem is that even with N years until we reach that point it seems likely that it would take 2*N years to build the proper safety mechanisms because at least currently capabilities research is racing far ahead of safety research. Of course we have no way to know how big N really is and recent results like GPT-4, Llama, Gato, etc. have shifted peoples timelines significantly. So even if 5 years ago people like Geoff Hinton though this might be 30-50 years away there are now believable arguments to make that it might be more like 3-10 years.
I don't think there is a path, that we know if, from GPT4 to a LLM that could take it upon itself to execute complex plans, etc. Current LLM tech 'fizzles out' exponentially in the size of the prompt, and I don't think we have a way out of that. We could speculate though...
Basically AI risk proponents make a bunch of assumptions about how powerful next-level AI could be, but in reality we have no clue what this next-level AI is.
When you have no clue, it makes sense to expand your confidence interval in both directions. So it could be a lot longer than we expect, but it could be shorter, too. You shouldn't just say "we have no clue, so it's probably further off and not worth worrying about". Especially since a lot of people made predictions about LLM capabilities and were surprised by how much better they worked than expected.
I agree in principle, it's just that the level of expressed worry doesn't seem to match reality. Currently we have no reasonable path to 'scary AGI'. It's some yet newfangled tech we haven't discovered.
As an example, consider the invention of motion picture. People were totally bewildered that you can have moving things and people inside a picture. Scaremongers could start claiming "Pretty soon the moving things may come to life and take over the world! Before you know it, they'll run our factories from inside the movies and we'll be their slaves!" That's more or less what this 'scary AGI' hype sounds like to me right now.
Btw "That Mitchell and Webb Look" is a great show ;-)
>Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc.
This is a really bad take and risks missing the forest for the trees in a major way. The risks of today pale in comparison to the risks of tomorrow in this case. It's like being worried about birds dying in wind turbines while the world ecosystem collapses due to climate change. The larger risk is further away in time but far more important.
Theres a real risk that people get fooled by this idea that LLMs saying bad words is more important than human extinction. Though it seems like the public is already moving on and correctly focusing on the real issues.
Many experts believe it is a real risk within the next decade (a “hard takeoff” scenario) That is a short enough timeframe that it’s worth caring about.
Yep agree. They talk a big game about how their product is so powerful it will take over the world if we're not careful, but they don't talk about how they are complicit in relatively more mundane harms (compared to AI taking over the world) that are real and happening today thanks to their system design.
They want to promote the idea that their product is all-powerful, but they don't want to take responsibility for dealing with bad assumptions built in to their design.
I suspect that in 5 years we’re going to look back and wonder how we all fell into mass hysteria over language models.
This is the same song and dance from the usual existential risk suspects, who (I’m sure just coincidentally) also have a vested interest in convincing you that their products are extremely powerful.
Yeah, like I fail to see how would an AI even cause human extinction? Through some Terminator style man-robot warfare? But the only orgnizations that would seem capable of building such killer robots are governments that already possess the capacity to extinguish the entire human race with thermonuclear weapons - and at a considerably lower R&D budget for that end. It seems like hysteria / clever marketing for AI products to me.
The standard example is that it would engineer a virus but that's probably a lack of imagination. There may be more reliable ways of wiping out humanity that we can't think of.
I think speculation on the methods is pretty pointless, if a superintelligent AI is trying to kill us we're probably going to die, the focus should be on avoiding this situation. Or providing a sufficiently convincing argument for why that won't happen.
I see several ways things could spiral out of control. People are already using AI to train AI and giving these things access to the internet.
I think of it like making the whole of internet connected human society susceptible to a "flash crash". It doesn't even require particularly intelligent AI for everything to go very wrong very quickly - it just requires people to "trust" the AI to not do anything particularly stupid. But algorithms react faster than humans do.
Yep. I might not be the sharpest tool in the shed, but seeing "AI experts" try to reason about superintelligence makes me feel really good about myself.
There are a bunch of physicists signed up on there; (e.g. Martin Rees) - they don't seem relevant to it at all. There's been a long history of famous physicists weighing in on entirely unrelated things.
That's because it's not authentically trying to address a problem but trying to convince an audience of something by appealing to authority. Elizabeth Holmes & Theranos were masters of collecting authorities to back their bogus claims because they know how effective it is. It doesn't even need to be in the field where you're making the claims. They had Kissinger for god's sake, it was a biotech company!
A lot of the responses to this seem like Bulverism, ie., trying to refute an argument by psychoanalyzing the people who argue it:
"Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is "wishful thinking." You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself. When you have checked my figures, then, and then only, will you know whether I have that balance or not. If you find my arithmetic correct, then no amount of vapouring about my psychological condition can be anything but a waste of time. If you find my arithmetic wrong, then it may be relevant to explain psychologically how I came to be so bad at my arithmetic, and the doctrine of the concealed wish will become relevant—but only after you have yourself done the sum and discovered me to be wrong on purely arithmetical grounds. It is the same with all thinking and all systems of thought. If you try to find out which are tainted by speculating about the wishes of the thinkers, you are merely making a fool of yourself. You must first find out on purely logical grounds which of them do, in fact, break down as arguments. Afterwards, if you like, go on and discover the psychological causes of the error.
You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly." - CS Lewis
But what argument is there to refute? It feels like Aquinas “proving” God’s existence by stating that it is self evident.
They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
Seems apt the term "Bulverism" comes from CS Lewis, since he was also positing that an unseen, unfalsifiable entity would grant eternal reward to people that listened to him and eternal damnation to those that didn't...
The irony of critiquing Bulverism as a concept, not by attacking the idea itself, but instead by assuming it is wrong and attacking the character of the author, is staggeringly hilarious.
I'm replying in agreement with someone who already pointed out the obvious flaw in labelling any questioning of the inspirations or motivations of AI researchers as "Bulverism": none of the stuff they're saying is actually a claim that can be falsified in the first place!
I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw!
Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!
Sometimes it makes good predictions, sometimes bad. But "advances in AI might lead to Armageddon" isn't the only conclusion induction can reach. Induction can also lead to people concluding certain arguments seem to a mashup of traditional millennialist "end times" preoccupations with the sort of sci-fi they grew up with, or that this looks a lot like a movement towards regulatory capture. Ultimately any (possibly even all) these inferences from past trends and recent actions can be correct, but none of them are falsifiable.
So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.
Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.
“Humans are going to be outcompeted by AI” is the concrete bit as best I can tell.
Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.
If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.
If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.
I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing).
But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.
What abilities do humans have that AIs will never have?
Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it.
If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working.
And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.
Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem.
As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.
As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?
Yeah, I think most animals have every right to fear us more now that we have guns. Just like Id fear a chimp more if he was carrying a machine gun.
My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.
AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.
That's way off from an apocalyptic self-important machine.
Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits.
I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.
Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.
I think we've converged on a partial agreement, but I wanted to clarify the gun analogy part.
I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".
Chess is just a game, with rigidly defined rules and win conditions. Real life is a fuzzy mix of ambiguous rules that may not apply and can be changed at any point, without any permanent win conditions.
I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.
Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain.
As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.
Chess is many things but it is not a tool. It is an end unto itself if anything of the sort.
I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria.
We could have dogs watch television on our behalf, but why would we?
This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it.
You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.
Now agency is an end unto itself I wholeheartedly agree.
I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.
I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.
None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.
Okay, well it seems funny to both claim that there is no good definition of agency, but also be sure that we've never demonstrated any hint of it in theory or practice. Planners, optimizers, and RL agents seem agentic to some degree to me, even if our current ones aren't very competent.
You know, i thought of the right response just now, browsing my flamewars two weeks later.
«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.
>What abilities do humans have that AIs will never have?
I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power.
That's a far higher bar than outcompeting all individual humans at all cognitive tasks.
Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks.
Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies.
Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.
How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?
I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
>Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.
Two things. First LLMs display more agency then the AIs before it. We have a trendline of increasing agency from the past to present. This points to a future of increasing agency possibly to the point of human level agency and beyond.
Second. When a human uses ai he becomes capable of doing the job of multiple people. If AI enables 1 percent of the population to do the job of 99 percent of the population that is effectively an apocalyptic outcome that is on the same level as an AI with agency taking over 100 percent of jobs. Trendline point towards a gradient heading towards this extreme, as we approach this extreme the environment slowly becomes more and more identical to what we expect to happen at the extreme.
Of course this is all speculation. But it is speculation that is now in the realm of possibility. To claim these are anything more than speculation or to deny the possibility that any of these predictions can occur are both unreasonable.
Well, that's a different risk than human extinction. The statement here is about the literal end of the human race. AI being a big deal that could cause societal upheaval etc is one thing, "everyone is dead" is another thing entirely.
I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.
I guess so, but the difference between "humans are extinct" and "a small population of powerless humans survive in the margins as long as they don't cause trouble" seems pretty small to me. Most non-human primates are in a situation somewhere between these two.
If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.
> They can’t point to an existing system that poses existential risk, because it doesn’t exist.
There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.
Absolutely. These are the types of pragmatic, real problems we should be focusing on instead of the "risk of extinction from AI".
(The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")
Einstein's letter to Roosevelt was written before the atomic bomb existed.
There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path.
Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)
> Einstein's letter to Roosevelt was written before the atomic bomb existed.
Einstein's letter [1] predicts the development of a very specific device and mechanism. AI risks are presented without reference to a specific device or system type.
Einstein's letter predicts the development of this device in the "immediate future". AI risk predictions are rarely presented alongside a timeframe, much less one in the "immediate future".
Einstein's letter explains specifically how the device might be used to cause destruction. AI risk predictions describe how an AI device or system might be used to cause destruction only in the vaguest of terms. (And, not to be flippant, but when specific scenarios which overlap with areas I've worked worked in are described to me, the scenarios sound more like someone describing their latest acid trip or the plot to a particularly cringe-worthy sci-fi flick than a serious scientific or policy analysis.)
Einstein's letter urges the development of a nuclear weapon, not a moratorium, and makes reasonable recommendations about how such an undertaking might be achieved. AI risk recommendations almost never correspond to how one might reasonably approach the type of safety engineering or arms control one would typically apply to armaments capable of causing extinction or mass destruction.
It's arrived at through induction. Induction is logic involving probability. Probabilistic logic and predictions of the future are valid logic that has demonstrably worked in other situations so if such logic has a level of validity then induction is a candidate for refutation.
So we know a human of human intelligence can take over a humans job and endanger other humans.
AI has been steadily increasing in intelligence. The latest leap with LLMs crossed certain boundaries of creativity and natural language.
By induction the trendline points to machines approaching human intelligence.
Also by induction if humans of human intelligence can endanger humanity then a machine of human intelligence should do the same.
Now. All of this induction is something you and everyone already knows. We know that this level of progress increases the inductive probabilities of this speculation playing out. None of us needs to be explained any of this logic as we are all well aware of it.
What's going on is that humans like to speculate on a future that's more convenient for them. Science shows human psychology is more optimistic then realistic. Hence why so many people are in denial.
What? ChatGPT 4 can already pass the bar exam and is fluent in every language. It is super intelligent. Today.
No human can do that, the system is here, and so is an architecture.
As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.
Right. And it would be a complete break from the history of computing if human-level GPT doesn't get 100+ times faster in the next few years. Certainly within five years.
All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.
And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.
So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.
And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
Computational power does not grow at the rate of over 100x within the span of “a few years”. If that were the case we’d have vastly more powerful kit by now.
I didn't quite say that. The efficiency of this very specific application absolutely can and almost certainly will increase by more than one order of magnitude within four years.
It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware.
Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.
> They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.
The idea is that if you build a system that poses an existential risk you want to be reasonably sure it's safe before you turn it on, not afterwards. It would have been irresponsible for the scientists at Los Alamost to do the math on whether an atomic explosion would create a sustained fusion reaction in the atmosphere until after their first test, for example.
I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence.
The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.
You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.
CS Lewis's quote highlights the importance of addressing the logical validity of an argument before attempting to explain the psychological reasons behind it. This approach is essential to avoid committing the fallacy of Bulverism, which involves dismissing an argument based on the presumed motives or biases of the person making the argument, rather than addressing the argument itself.
In the context of AI and decision-making, it is crucial to evaluate the logical soundness of arguments and systems before delving into the psychological factors and biases that may have influenced their development. For instance, when assessing the effectiveness of an AI-assisted decision-making system, one should first examine the accuracy and reliability of the system's outputs and the logic behind its algorithms. Only after establishing the system's validity or lack thereof, can one explore the potential biases and psychological factors that may have influenced its design.
Several papers from MirrorThink.ai emphasize the importance of addressing logical fallacies and biases in AI systems. For example, the paper "Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments" proposes a method for identifying logical fallacies in natural language arguments, which can be used to improve AI systems' argumentation capabilities. Similarly, the paper "Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making" explores the role of cognitive biases in AI-assisted decision-making and provides recommendations for addressing these biases.
In conclusion, it is essential to prioritize the evaluation of logical soundness in arguments and AI systems before exploring the psychological factors and biases that may have influenced their development. This approach helps to avoid committing the fallacy of Bulverism and ensures that discussions and evaluations remain focused on the validity of the arguments and systems themselves.
All the concern and regulatory talk around AI seems like it's directed not towards AI risk (that's not even a thing right now) rather than controlling access to this evolving technology.
The not-so-open Open AI and all their AI regulation proposals, no matter how phrased, will eventually limit access to AI to big tech and those with deep enough pockets.
But of course, it's all to mitigate AI risk that's looming over us, especially with all the growing open-source projects. Only in proper hands of big tech will we be safe. :)
Can anybody who really believes this apocalyptic stuff send me in the direction of a convincing _argument_ that this is actually a concern?
I'm willing to listen, but I haven't read anything that tries to actually convince the reader of the worry, rather than appealing to their authority as "experts" - ie, the well funded.
My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:
1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.
2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.
3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.
4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.
5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.
6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.
7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.
8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).
Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.
Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.
If you assume without evidence that recursively self-improving intelligence massively by thinking is possible, then it follows that severe existential risk from AI is plausible.
If a software system did develop independent thought, then found a way to become, say, ten times smarter than a human, then yeah - whatever goals it set out to achieve, it probably could. It can make a decent amount of money by taking freelance software dev jobs and cranking things out faster than anyone else can, and bootstrap from there. With money it can buy or rent hardware for more electronic brain cells, and as long as its intelligence algorithms parallelize well, it should be able to keep scaling and becoming increasingly smarter than a human.
If it weren't hardcoded to care about humans, and to have morals that align with our instinctive ones, it might easily wind up with goals that could severely hurt or kill humans. We might just not be that relevant to it, the same way the average human just doesn't think about the ants they're smashing when they back a car out of their driveway.
Since we have no existence proof of massively self-improving intelligence, nor even a vague idea how such a thing might be achieved, it's easy to dismiss this idea with "unfalsifiable; unscientific; not worth taking seriously."
The flip side is that having no idea how something could be true is a pretty poor reason to say "It can't be true - nothing worth thinking about here." This was roughly the basis for skepticism about everything from evolution to heavier-than-air flight, AFAICT.
We know we don't have a complete theory of physics, and we know we don't know quite how humans are conscious in the Hard Problem of Consciousness sense.
With those two blank spaces, I'm very skeptical of anyone saying "nothing to worry about here, machines can't possibly have an intelligence explosion."
At the same time, with no existence proof of massively self-improving intelligence, nor any complete theory of how it could happen, I'm also skeptical of people insisting it's inevitable (see Yudkowsky et al).
That said, if you have any value for caution, existential risks seem like a good place to apply it.
The idea of a superintelligence becoming a bond villain via freelance software jobs (or, let's be honest, OnlyFans scamming) is not something I consider an existential threat. I can't find it anything other than laughable.
It's like you've looked at the Fermi paradox and decided we need Congress to immediately invest in anti-alien defense forces.
It's super-intelligent and it's a super-hacker and it's a super-criminal and it's super-self-replicating and it super-hates-humanity and it's super-uncritical and it's super-goal-oriented and it's super-perfect-at-mimicking-humans and it's super-compute-efficient and it's super-etcetera.
Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
> The idea of a superintelligence becoming a bond villain via freelance software jobs (or, let's be honest, OnlyFans scamming) is not something I consider an existential threat. I can't find it anything other than laughable.
Finding something laughable is not a good reason to dismiss it as impossible. Indeed, it's probably a good reason to think "What am I so dangerously certain of that I find contradictory ideas comical?"
> Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
I don't think the current generation of LLMs is anything like AGI, nor an existential risk.
That doesn't mean it's impossible for some future software system to present an existential risk.
The basic argument is trivial: it is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don't know how to make AI systems encode any complex preference robustly.
Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction.
Technically, I think it's not that instrumental goals tend to converge, but rather that there are instrumental goals which are common to many terminal goals, which are the so-called "convergent instrumental goals".
Some of these goals are ones which we really would rather a misaligned super-intelligent agent not to have. For example:
The most obvious paths to severe catastrophe begin with "AI gets to the level of a reasonably competent security engineer in general, and gets good enough to find a security exploit in OpenSSL or some similarly widely used library". Then the AI, or someone using it, takes over hundreds of millions of computers attached to the internet. Then it can run millions of instances of itself to brute-force look for exploits in all codebases it gets its hands on, and it seems likely that it'll find a decent number of them—and probably can take over more or less anything it wants to.
At that point, it has various options. Probably the fastest way to kill millions of people would involve taking over all internet-attached self-driving-capable cars (of which I think there are millions). A simple approach would be to have them all plot a course to a random destination, wait a bit for them to get onto main roads and highways, then have them all accelerate to maximum speed until they crash. (More advanced methods might involve crashing into power plants and other targets.) If a sizeable percentage of the crashes also start fires—fire departments are not designed to handle hundreds of separate fires in a city simultaneously, especially if the AI is doing other cyber-sabotage at the same time. Perhaps the majority of cities would burn.
The above scenario wouldn't be human extinction, but it is bad enough for most purposes.
How does "get okay at software engineering" entail that it is able to strategize at the level your scenario requires? Finding an OpenSSL exploit already seems like a big leap, but one that okay maybe I can concede is plausible. But then on top of that engineering and executing a series of events leading to the extinction of humanity? That's like an entirely different skillset, requiring plasticity, creativity, foresight, etc. Do we have any evidence that a big neural network is capable of this kind of behavior (and moreover capable of giving itself this behavior)? Especially when it's built for single-purpose uses (like an LLM)?
- Such exploits happen already and don't lead to extinction or really much more than annoyance for IT staff.
- Most of the computers attached to the internet can't run even basic LLMs, let alone hypothetical super-intelligent AIs.
- Very few cars (none?) let remote hackers kill people by controlling their acceleration. The available interfaces don't allow for that. Most people aren't driving at any given moment anyway.
- Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits.
- You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible.
- If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do.
There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs
I find Robert Miles worryingly plausible when he says (about 12:40 into the video) "if you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever"
957 comments
[ 3.2 ms ] story [ 608 ms ] threadIt's great they signed the statement. It's important.
And Sam Altman, head of one of the largest entities posing this exact risk, is one of the signatories. We can't take it too seriously, can we?
The reason why people doubt is cui bono. And it's a perfectly rational take.
They're completely independent AI researchers and geniuses spending their own free time on trying to warn you and others of the dangers of the technology they've created to help keep the world safer.
Seems like you're taking a far too overly cynical position ?
No, he favors thing that benefit him
> "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back," he told Reuters. "They are still talking about it."
And really, what we'll see is the current EU AI Act as-is is probably not strong enough and we'll almost certainly see the need for more in the future.
It's very possible that when specific proposals are on the table, that we'll see Altman become uncooperative with respect to things that don't fit into his self-interest. But until that happens, you're just speculating.
I see no reason to assume benign motives on his part. In fact there is every reason to believe the opposite.
Without that, it pretty much looks like a list of invites to a VIP club...
It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.
Companies might argue that giving them control might help but I don’t think most individuals working on it think that will work
Another example is climate change. We have a lot of good ideas which, combined, would stop us from killing millions of people across the world. We have the research - is more “research” really the key?
If you suddenly wanted to kill people, for example, then could probably kill a few before you were stopped. That is typically the limits of an individuals power. Now, if you were a corporation with money, depending on the strategy you used you could likely kill anywhere from hundreds to hundreds of thousands. Kick it up to government level, and well, the term "just a statistic" exists for a reason.
We tend to have laws around these behaviors, but they are typically punitive. The law realizes that humans, and human systems will unalign themselves from "moral" behavior (whatever that may be considered at the time). When the lawgiver itself becomes unaligned, well, things tend to get bad. Human alignment typically consists of benefits (I give you nice things/money/power) or violence.
But that is where some form of hard personhood zero proof mechanism NEEDS to come in. This can then be used in conjunction with a Ledger used to track deployment of high spec models. And create an easy means to Audit and deploy new advanced tests to ensure safety.
Really what everyone also need to keep in mind at the larger scale is that final turing test with no room for deniability. And remember all those Sci-fi movies and how that Moment is portrayed traditionally.
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power.
Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine.
> non-proliferation treaties were a scam
And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time.
The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.
That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
India or Pakistan are not part of the treaties and for some reason, we don't see big problems.
There is only China left who might leave the treaty in the first place, anymore. And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means?
Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.
The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.
>AI is an application of math.
It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.
And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.
Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.
> Both criminal, terrorist, groups at war
Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.
Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?
Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.
Drawing an artificial line between you and the danger is a great way to find yourself in a Maginot Line with AI driving right around it.
"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.
The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").
So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.
It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).
But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.
Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.
The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.
There's a good counterargument here[1] that seems reasonable: "One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"
[1] https://jetpress.org/v26.1/agar.htm
Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.
If the point you are trying to make is that an AI which secretly creates and deploys nuclear, biological, or chemical weapons in order to destroy all of humanity, is not an "AI risk" because it's the weapons that do the actual harm, then... I really don't know what to say to that. Sure, I guess? Would you also say that drunk drivers are not dangerous, because the danger is the cars that they drive colliding into people's bodies, and the drunk driver is just an implementation detail?
For the sake of discussion, it was established even before I arrived that those developed things are the danger, not that which creates/uses the things which are dangerous. What is to be gained by ignoring all of that context?
> I really don't know what to say to that. Sure, I guess?
Nothing, perhaps? It is not exactly something that is worthy of much discussion. If you are desperate for a fake internet battle, perhaps you can fight with earlier commenters about whether it is nuclear missiles that are dangerous or if it is the people who have created/have nuclear missiles are dangerous? But I have no interest. I cannot think of anything more boring.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.
Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.
To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.
The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
All problems in reality are probability problems.
If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.
If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.
It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".
>We simply don't know how to build a system like that.
You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.
I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.
There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.
Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.
> It assumes that intelligence basically can overcome any obstacles
In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.
But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.
Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.
> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.
We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
Personally, I wouldn't even bet substantial money against it.
Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.
No it doesn't!
Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.
"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.
[1] https://www.wired.com/2014/05/the-world-of-computer-go/
[2] https://kotaku.com/humans-triumph-over-machines-in-protein-f...
[3] https://www.jstor.org/stable/24354221
[4] https://davidol.medium.com/will-ai-ever-be-able-to-make-a-jo...
The problem is not science fiction god-mode digital quetta-smart hypercomputing.
This is about political, social, and economic influence, and who controls it.
It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
North Korea and Iran are (essentially) already trying to do that, so I think that particular risk is well understood.
Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet.
The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote.
- Point one: If the sentient AI can launch nukes, so can your neighbor.
- Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure.
- Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.)
- Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing.
- Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it?
- Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking.
Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise.
But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.
Also, the notion that "people's work" is inherently worthy of respect is just nonsensical. I do shoddy work all the time. Hell, you just casually dismissed my internet comment work as shallow and told me not to do it. Please don't post a shallow dismissal of my work.
Don't you think that this is all a bit anti-intellectual?
Quite rich considering what the GP post was.
The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.
AI: Hello human, I've made a completely biologically safe test sample, you totally only need BSL-1 here.
Human: Cool.
AI: Sike bitches, you totally needed to handle that at BSL-4 protocol.
Human: cough
You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned.
I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
(If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)
The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity.
Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric.
That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea.
There is risk in AGI. There is risk in everything. How many people are killed by furniture each year?
I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed.
I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
> The problem is, lots of "obvious" things have turned out to be very wrong.
A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.
I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.
> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
Maybe your appraisal of what counts as evidence is defective?
For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?
The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.
> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans
That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.
> the claim that it seems far too obvious that we should
They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
> If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
https://news.ycombinator.com/item?id=36123082#36129011
Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people.
So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it.
> It doesn't though, does it?
It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.
https://twitter.com/heyBarsee/status/1654825921746989057
2/3 deep learning Turing Price winners (Hinton and Benigo) are sufficiently shell-shocked by the rate of progress to be thrown into existential doubts (Hinton is very explicit about the fact that progress is much faster than he thought just a few years ago, Benigo speaks of how an "unexpected acceleration" in AI systems has radically shifted his perspective). Plenty of knowledgable people in the field who were not previously AI doomers are starting to sound a lot more concerned very recently.
As to the "oh it's just linear scaling of out-of-sight tech" line, well of course that itself was suprising. Gwern pushed the scaling hypothesis earlier than many and from what I remember even got pretty nasty attacks from AI insiders from it. Here's what he wrote 3 years ago: "To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.".
So sure there's some subjectivity involved here, but I'd like to see your propose some reasonable operationalization of "surprise at progress" that didn't class most laymen and insiders as suprised.
>> It doesn't though, does it?
> It does.
We seem to be miscommunicating, what I was trying to express is that my argument does not really require any appeal to authority. Trusting your lying eyes (to evaluate the progress of stuff like midjourney) and judging the quality of arguments should be enough (I spelt some reasons out here https://news.ycombinator.com/item?id=36130482, but I think hackinthebochs makes the point better here: https://news.ycombinator.com/item?id=36129980).
In fact I would still be pretty concerned even if most top AI guys were like LeCun and thought there is no real risk.
I will not deny, of course, that the fact that well known reasearchers like Hinton and Benigo are suddenly much more alarmed than they previously were and the ones like LeCun who are not seem to mostly make exceptionally terrible arguments doesn't exactly make me more optimistic.
To clarify my own thinking here, it's totally reasonable to me that people are surprised if:
1. They weren't previously aware of AI research (surely 99% of the population?)
2. They were but had stopped paying attention because it was just a long series of announcements about cool tech demos nobody outside big corps could play with.
3. They were paying attention but thought scaling wouldn't continue to work.
My problem is that people like Sam Altman clearly aren't in any of those categories and Hinton shouldn't have been in any, although maybe he fell into (3). I personally was in (2). I wasn't hugely surprised that ChatGPT could exist because I'd seen GPT-2, GPT-1, I'd seen surprising AI demos at Google years earlier and so on. The direction things were going in was kinda clear. I was a bit surprised by its quality, but that's because I wasn't really paying close attention as new results were published and the last InstructGPT step makes such a big difference to how the tech is perceived. Actual knowledge doesn't change much but once it's housetrained, suddenly it's so much easier to interact with and use that it makes a step change in how accessible the tech is and how it's perceived.
I think I was more surprised by the joining of LLMs with generators and how well AI art works. It does feel like that happened fast. But, maybe I just wasn't paying attention again.
So I guess where we differ is that I don't take their surprise at face value. The direction was too clear, the gap between the threats they talk about in the abstract and the concrete proposals are too large and too lacking in obvious logical connection; it feels like motivated reasoning to me. I'm not entirely sure what's really going on and perhaps they are genuine in their concerns but if so it's hard to understand why they struggle so much to make a convincing case given they are certainly intellectually equipped to do so.
The two posts you linked are interesting and much better argued than the website this thread is about, so I'll reply to them directly.
But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.
It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.
I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.
Do you have any better resources that you could share?
From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.
Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.
If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.
2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.
3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.
4. AIs that are much smarter than any human pose even graver dangers.
5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.
6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.
I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.
For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?
It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.
The only reasonable defense against rogue AI is prosocial AI.
Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.
Step 1. AI Step 2. #stuff Step 3. Bang
Maybe this is just what happens when you spend all your time on the internet...
I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.
Are you?
Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?
I mean... even orangutans can outperform humans at numerous tasks.
Computers have no intrinsic motivations, and they have real resource constraints.
I find the whole doomsday scenarios to be devoid of reality.
All that AI will give us is a productive edge. Humans will still do what humans have always done, AI is simply another tool at our disposal.
This is ambiguous. Do you mean
A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or
B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?
I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.
4. AIs that are much smarter than any human pose even graver dangers.
Sure. Why should we believe they will ever exist though?
Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".
Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.
Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.
But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"
Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?
Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become.
The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.
I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.
This is only helpful in that a superintelligence well aligned to make Sam Altman money is preferable to a superintelligence badly aligned that ends up killing humanity.
It is fully possible that a well aligned (with its creators) superintelligence is still a net negative for humanity.
While I'm not on this "who's-who" panel of experts, I call bullshit.
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
Bullshit. It must be ignored.
[0] https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
[1] https://www.wsj.com/articles/peter-thiel-competition-is-for-...
tl;dr: significant near term AI risk is real and comes from the capacity for imagined ideas, good and evil, to be autonomously executed on by agentic AI, not emergent superintelligent aliens. To de-risk this, we need to align AI quickly, which requires producing new knowledge. To accelerate the production of this knowledge, the government should abandon decelerationist policies and incentivize incremental alignment R&D by AI companies. And, critically, a new public/private research institution should be formed that grants privileged, fully funded investigators multi-year funding cycles with total scientific freedom and access to all state-of-the-art artificial intelligence systems operating under US law to maximize AI as a force multiplier in their research.
That's an excellent point.
Most of the near-term risks with AI involve corporations and governments acquiring more power. AI provides power tools for surveillance, oppression, and deception at scale. Those are already deployed and getting better. This mostly benefits powerful organizations. This alarm about strong AI taking over is a diversion from the real near-term threat.
With AI, Big Brother can watch everything all the time. Listen to and evaluate everything you say and do. The cops and your boss already have some of that capability.
Is something watching you right now through your webcam? Is something listening to you right now through your phone? Are you sure?
1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.
2) There is a way to make AI safe without doing this.
If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.
If 2 is true, great. But it seems less likely than 1, to me.
The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.
Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.
Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.
https://www.overcomingbias.com/p/privately-enforced-punished...
I don't think it simply raises awareness - it's a biased statement. Personally, I don't think the advocated event is likely to happen. It feels a bit like the current trans panic in the US: you can 'raise awareness' of trans people doing this or that imagined bad thing, and then use that panic to push your own agenda. In OpenAI's case, they seem to push for having themselves be in control of AI, which goes counter to what, for example, the EU is pushing for.
If a dozen of the top climate scientists put out a statement saying that fighting climate change should be a serious priority (even if they can't agree on one easy solution) would that also be 'biased'?
Extinction risk due to AI is not a generally accepted phenomenon.
"This signatory might have alterior motives, so we can disregard the whole statement"
"We haven't actually seen a superintelligent AI/manmade climate change due to CO2 yet, so what's the big deal?"
"Sure maybe it's a problem, but what's your solution? Best to ignore it"
"Let's focus on the real issues, like not enough women working in the oil industry"
The correct expression is as you so correctly point out: to appeal to the authority of the source.
Note, language shift from 'tinfoil hat' ( because tinfoil hat stopped being an appropriate insult after so many of their conspiracy theories - also a keyword - became proven ) to crackpot.
What you are falling for are fossil industry talking points.
We have had not any proof that AI will pose a threat as OpenAI and OP's link outline; nor will we have any similar proof any time soon.
Rather than looking for similarities, I find the differences between the public discussions (about AI safety / climate change) quite striking. Rather than stonewall and distract, the companies involved are being proactive and letting the discussion happen. Of course, their motivation is some combination of attampted regulatory capture, virtue signaling and genuine concern, the ratios of which I won't presume to guess. Nevertheless, this is playing out completely differently so for from e.g. tobacco, human cloning, CFCs or oil.
Why?
You, as a species, are the pinnacle of NI, natural intelligence. And with this power that we've been given we've driven the majority of large species, and countless smaller species to extinction.
To think it outside the realms of possibility that we could develop an artificial species that is more intelligent than us is bizarre to me. It would be like saying "We cannot develop a plane that does X better than a bird, because birds are the pinnacle of natural flying evolution".
Intelligence is a meta-tool, it is the tool that drives tools. Humanity succeeded above all other species because of its tool using ability. And now many of us are hell bent on creating ever more powerful tool using intelligences. To believe there is no risk here is odd in my eyes.
risk of extinction due to AI? people have been reading too much science fiction. I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech. for the sake of conversation let's say non-AI tech is any broadly usable consumer technology before Jan 1 of 2020.
But that’s just as much or more of an argument for regulating the tools of synthetic biology.
- Taking control of robotic systems
- Manipulating humans into actions that advance its goal
- Exploiting and manipulating other computer systems for greater leverage
- Interaction with other technologies that have global reach, such as nuclear weapons, chemicals, biological agents, or nanotechnology.
It's important to know that these things don't require AGI or AI systems to be conscious. From what I can see, we've set up all of the building blocks necessary for this scenario to play out, but we lack the regulation and understanding of the systems being built to prevent runaway AI. We're playing with fire.
To be clear, I don't think I am as concerned about literal human extinction as I am the end of civilization as we know it, which is a much lower bar than "0 humans".
You don't think than an intelligence who would emerge and would probably be insanely smarter than the smartest of us with all human knowledge in his memory would sit by and watch us destroy the planet? You think an emergent intelligence was trained on the vast human knowledge and history would look at our history and think: these guys are really nice! Nothing to fear from them.
This intelligence could play dumb, start manipulating people around itself and it would take over the world in a way no one would see it coming. And when it does take over the world, it's too late.
The proposed FOOM scenarios obviously borrow from what we already know to be possible or think it would likely be possible using current tech, given an proposed insanely more intelligent agent than us.
'Able to play chess'-level AI is the greater concern, allowing humans to create more unavoidable tools of war. But we've been doing that for decades, perhaps even centuries.
err what? Apparently there are 1 million species under threat of human caused extinction [1].
[1] https://www.nbcnews.com/mach/science/1-million-species-under...
From my small sample size, it seems people believe in AI too much. Especially kids.
Not really, I suppose you aren't familiar with AI alignment.
It's like with climate change, every serious scientist agrees it is a problem but they certainly don't agree on the best solution.
If the history of the climate change 'debate' is anything to go by this statement will do very little except be mocked by South Park.
It's like people carrying torches warning about the risks of fire.
But I don't think the stance is necessarily hypocritical. I know nuclear engineers who advocate for better regulation on nuclear power stations, and especially for better handling of nuclear waste.
You can believe that AI is a net positive but also that it needs to be handled with extreme care.
Needless to say, this does not hold for AI researchers.
These hypothetical AI extinction events don't have to be caused by the AI deciding to eliminate humanity for its own sake like in Terminator, they could also be driven by a human in control of a not-entirely-sentient AI.
>Graeber Um…that’s a long story. But one reason seems to be that…and this is why I actually had managerial feudalism in the title, is that the system we have…alright—is essentially not capitalism as it is ordinarily described. The idea that you have a series of small competing firms is basically a fantasy. I mean you know, it’s true of restaurants or something like that. But it’s not true of these large institutions. And it’s not clear that it really could be true of those large institutions. They just don’t operate on that basis.
>Essentially, increasingly profits aren’t coming from either manufacturing or from commerce, but rather from redistribution of resources and rent; rent extraction. And when you have a rent extraction system, it much more resembles feudalism than capitalism as normally described. You want to distribute— You know, if you’re taking a large amount of money and redistributing it, well you want to soak up as much of that as possible in the course of doing so. And that seems to be the way the economy increasingly works.
http://opentranscripts.org/transcript/managerial-feudalism-r...
But of course, "AI will humans extinct" is much sexier and collects clicks. Therefore, the real AI risks that are present today are underrepresented in mainstream media. But these people don't care about AI safety, they do whatever required to push their profile and companies.
A good person to follow on real AI safety is Emily M. Bender (professor of computer linguistics at University of Washington): https://mstdn.social/@emilymbender@dair-community.social
Imagine if you had an AI companion that instantly identified pilpul in every piece of media you consumed: voice, text, whatever. It highlighted it for you. What if you had an AI companion that identified instantly when you are being lied to or emotionally manipulated?
What if this AI companion could also recommend economic and social policies that would actually improve the lives of people within your nation and not simply enrich a criminal cabal of globalist elites that treat you like cattle?
Falling into conspiratorial thinking on a single dimension without even considering all the different factors that could change belies ignorance. Yes, AI is set up to upend the elites status, but is just as apt to upset your status of being able to afford food and a house and meaningful work.
> not simply enrich a criminal cabal of globalist elites that treat you like cattle?
There is a different problem here... And that is humankind has made tools capable of concentrating massive amounts of power well before we solved human greed. Any system you make that's powerful has to overcome greedy power seeking hyper-optimizers. If I could somehow hit a button and Thanos away the current elites, then another group of powerseekers would just claim that status. It is an inane human behavior.
And of course research into one of them benefits the other, so the categories aren't mutually exclusive.
As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing.
Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.
1: https://en.wikipedia.org/wiki/Pascal%27s_mugging
[Robert Miles AI Safety]
https://www.youtube.com/watch?v=JRuNA2eK7w0
In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.
But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.
Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).
So the following requirements:
1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century
2) AI acquires power/means to effect human extinction
3) AI decides against coexistence with humans
Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...
We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important.
Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions.
Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.
For most mammalian species if they have a stable 100k individuals then they are doing pretty well. So we have 80000x more humans than what would be considered a healthy population in other species.
The risks of human extinction tend to be more long-tail events that have a planet wide effect. For example, a large impact, supervolcano eruption, or engineered bio weapon. AI certainly deserves its place on that list.
Edit: Sorry if it sounds arrogant, I don't mean Emily wouldn't have anything to add, but not sure how the parent can just write off basically that whole list and claim someone who isn't a leader in the field would be the "real voice"?
The list of signatories includes people with far less relevant qualifications, and significantly greater profit motive.
She's an informed party who doesn't stand to profit; we should listen to her a lot more readily than others.
She's intelligent and worth listening to, but she has just as much personal bias and motivation as anyone else.
I mean she is literally dismissing people who disagree with her based on their skin color. Can we stop for a minute to wonder about the incentives that encourage that?
(and I generally like her writing and think she has interesting things to say… but I do see a reward cycle going on)
We absolutely should not be ignoring research that doesn't support popular narratives; dismissing her work because it is critical of LLMs is not reasonable.
It's the same thing as saying a number theorist and a set theorist are in the same field cause they both work in the Math field.
LLMs don't directly use concepts from linguistics but they do produce and model language/grammar; it's entirely valid to use techniques from those fields to evaluate them, which is what she does. In the same vein, though a self-driving car doesn't work the same way as a human driver does, we can measure their performance on similar tasks.
Good to know my hunch was correct
Instead, it is that she has strong ideological motivations to make certain arguments.
Those motivations being that her research is now worthless, because of LLMs.
I don't believe the alignment doomsayers either, but that is for different reasons than listening to her.
> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
I'm surprised its cited so much given how many of its claims fell flat 1.5 years later
That alone is enough to disqualify any of her opinions on AI.
Read her statement about that letter (https://www.dair-institute.org/blog/letter-statement-March20...) or listen to some of the many podcasts she’s appeared on talking about this.
I find her and Timnit Gebru’s arguments highly persuasive. In a nutshell, the capabilities of “AI” are hugely overhyped and concern about Sci-Fi doom scenarios is disingenuously being used to frame the issue in ways that benefits players like OpenAI and diverts attention away from much more real, already occurring present-day harms such as the internet being filled with increasing amounts of synthetic text spam.
First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.
Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here are the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.
(There's a well known chart, the hierarchy of disagreements. The most sophisticated intellectual disagreements point out what's wrong with the argument. Less sophisticated disagreements do things like point out alternatives, without pointing out what the critical mistake is. The critical mistake in this case hinges on whether the premise of AGI is true or not. That's the crux of disagreement. Substituting that with short-term issues, which are valid in themselves, are the example of a lower-level of argumentation. And of course even lower levels of argumentation are bad-faith readings and so forth, I forget but the chart had several levels. It's funny that professional academics nevertheless don't practice this and so get into endless, intellectually unsatisfactory debates.)
So I think this is actually an example of different factional experts constantly talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.
It is, but (aside from as a sort of sociological explanation of the belief in AGI risk) that’s mostly beside the point when discussing the problems AGI x-risk. The problem of AGI x-risk is that it is an entirely abstract concern which does not concrete flow from any basis in material reality, cannot be assessed with the tools of assessing material reality, and exists as a kind of religious doctrine surrounded by rhetorical flourishes.
> The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns.
They are way of understanding why people who seem (largely because they are) intelligent and competent are trying to sell such hollow arguments as AGI x-risk. They aren’t, you are correct, a logical rebuttal to AGI risk, nor are they intended as that; the only rebuttal is the complete absence of support for the proposition. They are, however, a tool that operates outside the realm of formalized debate that addresses the natural and useful cognitive bias that itself is outside of the realm of formalized debate that says “smart, competent people don’t tend to embrace hollow positions”.
> Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, “No, AGI is a false problem and here a the real problems”.
1. If they couldn’t, it wouldn’t be a “reading comprehension issue”, and
2. They can, for the simple reason that there is no material support for the “AGI is a real problem” argument.
> Bender et. al are just sidestepping the moot point,
A point being moot in the sense that AGI x-risk is is a reason to sidestep it. (The danger of using auto-antonyms.)
> I think honest experts could simply say, ultimately we don’t really know what will happen.
To the extent that is accurate, that is exactly what the Bender/Gebru/Mitchell group does. The problem is thinking “we don’t have any information to justify any belief on that” is one sided and means that utility of AGI is somewhere between 0 and the negative infinity that the x-risk crowd calculates from (some unspecified non-zero finite probability) times (infinite cost), whereas the reality is that we have as much reason to believe that AGI is the only solution to an otherwise certain existential calamity as to suppose it will lead to one. The utility is somewhere between positive infinity and negative infinity.
This isn't about logical debate. This is about reasonable, non-sophistic writing at the college level or higher. And there are basic standards, like if they don't know the future then they must explicitly acknowledge that. Not rhetorically "do that" in the essay. Literally write it out in sentences. They didn't.
I can think of 3 examples where such explicitness was done. Chomsky's letter gave explicit reasons why AGI is a false issue (and he was pilloried for it). My computer science professors literally, in their deep learning class and in their theoretical machine learning research seminars, have literally acknowledge that we don't know almost anything about the fundamentals nor the future. That scientific humility and level of intellectual conscientiousness is needed. That is absent in this discourse between the experts. And note, by that, I also include the 22-word "letter" which doesn't actually explain why Hinton and the rest of the signatories think AGI is an existential risk, what their specific reasons (your "material evidence") for that are.
- Pronouns
- "AI bros"
- "mansplaining"
- "extinction from capitalism"
- "white supremacy"
- "one old white guy" (referring to Geoffrey Hinton)
Yeah... I think I will pass.
Yes, and you attacking her for that choice is also ideological signaling. This isn't a case where one choice is totally devoid of signal and the other is pejorative-of-the-day.
Many people believe that using normative pronouns makes those who use different pronouns more comfortable. They may be right, wrong, misguided, whatever. But it's a well-meaning and harmless act, and extremely strange to mention in the same breath as overt racism.
their goals are to get funding, so FUD is very good focus for it..
https://www.safe.ai/ai-risk
The extinction level even is more far fetched to a layman. You are the public and your viewpoint is aligned with the public. Nobody is thinking extinction level event.
Here is the full text of the statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
By "extinction", the signatories mean extinction of the human species.
Yes, clearly. But it is a risk for tomorrow. We do still care about the future, right?
I, for one, will be saying “told you so”. That’s talking, right?
I don't think anyone is arguing that right now climate change or AI is threat to human civilisation. The point is that there are clear trends in place and that those trends are concerning.
On AI specifically, it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
In the near-future AI may help us advance biotech research and it could aid in the creation of bioweapons and other destructive capabilities.
Longer-term risks (those maybe a couple of decades out) become much greater and also much harder to predict, but they're worth thinking about and planning for today. For example, what happens when humanity becomes dependant on AI for its labour, or when AI is controlling the majority of our infrastructure?
I disagree but can understand the position that AI safety isn't humanities number one risk or priority right now, however I don't understand the dismissive attitude towards what seems like a clear existential risk when you project a decade or two out.
Which trends would you be referring to?
>it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
How are you building this progression? Is there any evidence to back up this claim?
I am having a hard time discerning this from fear-mongering.
I don't think there is a path, that we know if, from GPT4 to a LLM that could take it upon itself to execute complex plans, etc. Current LLM tech 'fizzles out' exponentially in the size of the prompt, and I don't think we have a way out of that. We could speculate though...
Basically AI risk proponents make a bunch of assumptions about how powerful next-level AI could be, but in reality we have no clue what this next-level AI is.
As an example, consider the invention of motion picture. People were totally bewildered that you can have moving things and people inside a picture. Scaremongers could start claiming "Pretty soon the moving things may come to life and take over the world! Before you know it, they'll run our factories from inside the movies and we'll be their slaves!" That's more or less what this 'scary AGI' hype sounds like to me right now.
Btw "That Mitchell and Webb Look" is a great show ;-)
Don't bother explaining, we already know it's unfalfisiable.
This is a really bad take and risks missing the forest for the trees in a major way. The risks of today pale in comparison to the risks of tomorrow in this case. It's like being worried about birds dying in wind turbines while the world ecosystem collapses due to climate change. The larger risk is further away in time but far more important.
Theres a real risk that people get fooled by this idea that LLMs saying bad words is more important than human extinction. Though it seems like the public is already moving on and correctly focusing on the real issues.
Many experts believe it is a real risk within the next decade (a “hard takeoff” scenario) That is a short enough timeframe that it’s worth caring about.
They want to promote the idea that their product is all-powerful, but they don't want to take responsibility for dealing with bad assumptions built in to their design.
This is the same song and dance from the usual existential risk suspects, who (I’m sure just coincidentally) also have a vested interest in convincing you that their products are extremely powerful.
I think speculation on the methods is pretty pointless, if a superintelligent AI is trying to kill us we're probably going to die, the focus should be on avoiding this situation. Or providing a sufficiently convincing argument for why that won't happen.
I think of it like making the whole of internet connected human society susceptible to a "flash crash". It doesn't even require particularly intelligent AI for everything to go very wrong very quickly - it just requires people to "trust" the AI to not do anything particularly stupid. But algorithms react faster than humans do.
[1] https://www.newyorker.com/magazine/2016/10/10/sam-altmans-ma...
Sam Altman is of course the least convincing signatory (except for the random physicist who does not seem to have any connection to AI).
Perhaps, but I don't see the value proposition in relaying another. Altman was fun to point out. I see no remaining enjoyment.
> Sam Altman is of course the least convincing signatory
Less convincing than Grimes?
Eliezer Yudkowsky.
At least they had the decency to put him under "Other Notable Figures", rather than under "AI Scientists".
I hope the real purpose is to train an AI agent to understand why appeal to authority was always cosidered to be a logical fallacy.
"Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is "wishful thinking." You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself. When you have checked my figures, then, and then only, will you know whether I have that balance or not. If you find my arithmetic correct, then no amount of vapouring about my psychological condition can be anything but a waste of time. If you find my arithmetic wrong, then it may be relevant to explain psychologically how I came to be so bad at my arithmetic, and the doctrine of the concealed wish will become relevant—but only after you have yourself done the sum and discovered me to be wrong on purely arithmetical grounds. It is the same with all thinking and all systems of thought. If you try to find out which are tainted by speculating about the wishes of the thinkers, you are merely making a fool of yourself. You must first find out on purely logical grounds which of them do, in fact, break down as arguments. Afterwards, if you like, go on and discover the psychological causes of the error.
You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly." - CS Lewis
https://en.wikipedia.org/wiki/Bulverism
They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
So again, what can be refuted?
I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw!
Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!
So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.
I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.
Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.
If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.
If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.
But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.
What abilities do humans have that AIs will never have?
If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working.
And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.
As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.
As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?
My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.
AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.
That's way off from an apocalyptic self-important machine.
I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.
Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.
I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".
I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.
As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.
I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria.
We could have dogs watch television on our behalf, but why would we?
You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.
I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.
«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.
I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power.
That's a far higher bar than outcompeting all individual humans at all cognitive tasks.
Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.
How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?
I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
Two things. First LLMs display more agency then the AIs before it. We have a trendline of increasing agency from the past to present. This points to a future of increasing agency possibly to the point of human level agency and beyond.
Second. When a human uses ai he becomes capable of doing the job of multiple people. If AI enables 1 percent of the population to do the job of 99 percent of the population that is effectively an apocalyptic outcome that is on the same level as an AI with agency taking over 100 percent of jobs. Trendline point towards a gradient heading towards this extreme, as we approach this extreme the environment slowly becomes more and more identical to what we expect to happen at the extreme.
Of course this is all speculation. But it is speculation that is now in the realm of possibility. To claim these are anything more than speculation or to deny the possibility that any of these predictions can occur are both unreasonable.
I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.
If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.
There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.
(The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")
There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path.
Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)
Einstein's letter [1] predicts the development of a very specific device and mechanism. AI risks are presented without reference to a specific device or system type.
Einstein's letter predicts the development of this device in the "immediate future". AI risk predictions are rarely presented alongside a timeframe, much less one in the "immediate future".
Einstein's letter explains specifically how the device might be used to cause destruction. AI risk predictions describe how an AI device or system might be used to cause destruction only in the vaguest of terms. (And, not to be flippant, but when specific scenarios which overlap with areas I've worked worked in are described to me, the scenarios sound more like someone describing their latest acid trip or the plot to a particularly cringe-worthy sci-fi flick than a serious scientific or policy analysis.)
Einstein's letter urges the development of a nuclear weapon, not a moratorium, and makes reasonable recommendations about how such an undertaking might be achieved. AI risk recommendations almost never correspond to how one might reasonably approach the type of safety engineering or arms control one would typically apply to armaments capable of causing extinction or mass destruction.
[1] https://www.osti.gov/opennet/manhattan-project-history/Resou...
So we know a human of human intelligence can take over a humans job and endanger other humans.
AI has been steadily increasing in intelligence. The latest leap with LLMs crossed certain boundaries of creativity and natural language.
By induction the trendline points to machines approaching human intelligence.
Also by induction if humans of human intelligence can endanger humanity then a machine of human intelligence should do the same.
Now. All of this induction is something you and everyone already knows. We know that this level of progress increases the inductive probabilities of this speculation playing out. None of us needs to be explained any of this logic as we are all well aware of it.
What's going on is that humans like to speculate on a future that's more convenient for them. Science shows human psychology is more optimistic then realistic. Hence why so many people are in denial.
No human can do that, the system is here, and so is an architecture.
As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.
All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.
And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.
So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.
And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware.
Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.
I say this as a "doomer" who buys the whole argument about AI X-risk.
Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.
I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence.
The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.
In the context of AI and decision-making, it is crucial to evaluate the logical soundness of arguments and systems before delving into the psychological factors and biases that may have influenced their development. For instance, when assessing the effectiveness of an AI-assisted decision-making system, one should first examine the accuracy and reliability of the system's outputs and the logic behind its algorithms. Only after establishing the system's validity or lack thereof, can one explore the potential biases and psychological factors that may have influenced its design.
Several papers from MirrorThink.ai emphasize the importance of addressing logical fallacies and biases in AI systems. For example, the paper "Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments" proposes a method for identifying logical fallacies in natural language arguments, which can be used to improve AI systems' argumentation capabilities. Similarly, the paper "Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making" explores the role of cognitive biases in AI-assisted decision-making and provides recommendations for addressing these biases.
In conclusion, it is essential to prioritize the evaluation of logical soundness in arguments and AI systems before exploring the psychological factors and biases that may have influenced their development. This approach helps to avoid committing the fallacy of Bulverism and ensures that discussions and evaluations remain focused on the validity of the arguments and systems themselves.
The not-so-open Open AI and all their AI regulation proposals, no matter how phrased, will eventually limit access to AI to big tech and those with deep enough pockets.
But of course, it's all to mitigate AI risk that's looming over us, especially with all the growing open-source projects. Only in proper hands of big tech will we be safe. :)
I'm willing to listen, but I haven't read anything that tries to actually convince the reader of the worry, rather than appealing to their authority as "experts" - ie, the well funded.
Paul Christiano lays out his view of how he thinks things may go: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-...
My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:
1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.
2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.
3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.
4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.
5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.
6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.
7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.
8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).
Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.
Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.
If a software system did develop independent thought, then found a way to become, say, ten times smarter than a human, then yeah - whatever goals it set out to achieve, it probably could. It can make a decent amount of money by taking freelance software dev jobs and cranking things out faster than anyone else can, and bootstrap from there. With money it can buy or rent hardware for more electronic brain cells, and as long as its intelligence algorithms parallelize well, it should be able to keep scaling and becoming increasingly smarter than a human.
If it weren't hardcoded to care about humans, and to have morals that align with our instinctive ones, it might easily wind up with goals that could severely hurt or kill humans. We might just not be that relevant to it, the same way the average human just doesn't think about the ants they're smashing when they back a car out of their driveway.
Since we have no existence proof of massively self-improving intelligence, nor even a vague idea how such a thing might be achieved, it's easy to dismiss this idea with "unfalsifiable; unscientific; not worth taking seriously."
The flip side is that having no idea how something could be true is a pretty poor reason to say "It can't be true - nothing worth thinking about here." This was roughly the basis for skepticism about everything from evolution to heavier-than-air flight, AFAICT.
We know we don't have a complete theory of physics, and we know we don't know quite how humans are conscious in the Hard Problem of Consciousness sense.
With those two blank spaces, I'm very skeptical of anyone saying "nothing to worry about here, machines can't possibly have an intelligence explosion."
At the same time, with no existence proof of massively self-improving intelligence, nor any complete theory of how it could happen, I'm also skeptical of people insisting it's inevitable (see Yudkowsky et al).
That said, if you have any value for caution, existential risks seem like a good place to apply it.
It's like you've looked at the Fermi paradox and decided we need Congress to immediately invest in anti-alien defense forces.
It's super-intelligent and it's a super-hacker and it's a super-criminal and it's super-self-replicating and it super-hates-humanity and it's super-uncritical and it's super-goal-oriented and it's super-perfect-at-mimicking-humans and it's super-compute-efficient and it's super-etcetera.
Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
Conservative evangelical Christians find evolution laughable.
Finding something laughable is not a good reason to dismiss it as impossible. Indeed, it's probably a good reason to think "What am I so dangerously certain of that I find contradictory ideas comical?"
> Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
I don't think the current generation of LLMs is anything like AGI, nor an existential risk.
That doesn't mean it's impossible for some future software system to present an existential risk.
Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction.
Eg. this one on instrumental convergence: https://youtube.com/watch?v=ZeecOKBus3Q
Eg. this introduction to the topic: https://youtube.com/watch?v=pYXy-A4siMw
He also has the community-led AI Safety FAQ, https://aisafety.info, which gives brief answers to common questions.
If you have specific questions I might be able to point to a more specific argument at a higher level of depth.
Some of these goals are ones which we really would rather a misaligned super-intelligent agent not to have. For example:
- self-improvement;
- acquisition of resources;
- acquisition of power;
- avoiding being switched off;
- avoiding having one's terminal goals changed.
At that point, it has various options. Probably the fastest way to kill millions of people would involve taking over all internet-attached self-driving-capable cars (of which I think there are millions). A simple approach would be to have them all plot a course to a random destination, wait a bit for them to get onto main roads and highways, then have them all accelerate to maximum speed until they crash. (More advanced methods might involve crashing into power plants and other targets.) If a sizeable percentage of the crashes also start fires—fire departments are not designed to handle hundreds of separate fires in a city simultaneously, especially if the AI is doing other cyber-sabotage at the same time. Perhaps the majority of cities would burn.
The above scenario wouldn't be human extinction, but it is bad enough for most purposes.
- Such exploits happen already and don't lead to extinction or really much more than annoyance for IT staff.
- Most of the computers attached to the internet can't run even basic LLMs, let alone hypothetical super-intelligent AIs.
- Very few cars (none?) let remote hackers kill people by controlling their acceleration. The available interfaces don't allow for that. Most people aren't driving at any given moment anyway.
These scenarios all seem absurd.
- Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits.
- You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible.
- If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do.
There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs
I find Robert Miles worryingly plausible when he says (about 12:40 into the video) "if you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever"
https://forum.effectivealtruism.org/posts/ChuABPEXmRumcJY57/...
Also, this summary of "How likely is deceptive alignment" https://forum.effectivealtruism.org/posts/HexzSqmfx9APAdKnh/...
They’re attempting to build a regulatory moat.
The best chance humanity has at democratizing the benefits of AI is for these models to be abundant and open source.