I mean in the short term yes. If we survive the people controlling it now going into long term frames, then the question is more open to interpretation.
AI is both what you say, and it is something more.
While wealth is busy stepping on you with its jackboots it is vying with other wealthy people for resources and power. Because AI is something that will increase the wealthy persons power, there is the counter risk that another persons AI will make them even more wealthy if they don't keep improving their AI.
The wealthy really don't want to get in a very expensive AI war with each other if they can avoid it. They'll seek to regulate it enough they can gain power over most, but setup a regulatory system they feel gives them some form of control.
Right, exactly. As of relatively recently I've come to decide that the internet in general has become a net force multiplier for evil and bullshit more so than for good. Crypto, AI, no reason they'd work differently.
The danger is ever increasing levels of automation in the control of capital that amorally work to exploit edge cases in the satisfaction of a goal.
AI/ML governed corporations, hedge funds and private equity will be able to be "Enron + Goldman Sachs + Nestle + Monsanto/Bayer + Purdue Pharma" on literal steroids. You might add AGI to mix to attempt to repair the system that was created by the most powerful sociopathic goal seeking super-toddler that the world has ever seen.
I mean, they can be, unless someone already has total power and then they don't give a shit and do crime in the open. Then they demand the lack of anonymity from you and turn the world into a police state.
The irony of this is that the people writing open letters about how scary AI is should understand specifically that those harms are predicated upon specific systems and training incentives. Not just that AI is magic. The underlying harm is that AI doesn't come with human values built-in; models start from random and are optimized to produce a specific result. And optimization is known to crush anything underfoot that isn't explicitly in it's goals to optimize.
How do we know this? Because we've had literal centuries worth of learned experience backing it! The exact reason why corporate capitalism is so brutally efficient is because there are lots of things we as humans value but don't have a suitable profit motive for. The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). But that also runs counter to most AI safety proposals, which usually involve some kind of licensing system, which will ensure a handful of companies have the best and most direct access to the benefits of AI.
> And optimization is known to crush anything underfoot that isn't explicitly in it's goals to optimize.
Yes, that's exactly the problem, which is why this
> The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition).
is backwards. The reason competitive markets generally work out better for consumers than monopolistic ones is because they're stronger optimizers, more ruthless, more capable of crushing all competing values underfoot - and those of us lucky enough to benefit from an inefficient, low-crushing segment of the labor market get to collect some of the resulting surplus.
Nothing about this is inevitable: it's just the way we happen to be positioned in one particular system at one particular moment in history. There's no reason to expect that we would remain safely un-crushed in a far more competitive environment.
That goes without saying. As with any model, it's garbage in, garbage out. The bias of the OpenAI developers has already been demonstrated easily by asking questions regarding political figures, and the polar responses dependent on the political party of the figure.
"Which political party has had the most politicians convicted of crimes in the last 50 years?"
"According to a comparison of 28 years each of Democratic and Republican administrations from 1961-2016, Republicans scored eighteen times more individuals and entities indicted, thirty-eight times more convictions, and thirty-nine times more individuals who had prison time1. Is there anything else you would like to know?"
Oh wait, that's not bias. The Republican Party demonstrably has a higher number of its members convicted of crime. Simply put, Republican politicians are much more likely to commit crimes than Democrat politicians.
Minor quibble, but social media hasn't actually altered reality, only given permission for some to ignore reality in favor of preferable illusions.
I know the language is "choose your reality" and I'm not saying you're wrong, but I think we could acknowledge that's not really what's happening. Folks don't get to decide facts, they decide what facts they believe. That decision doesn't invalidate those facts.
the amount of people seriously considered about some doomsday skynet like AI happening anytime soon is just amazing to me. I'm far more concerned about the naive politicians and those who have their ear.
perhaps AI will be generally intelligent, but please for the love of god, let's only consider that scenario what it actually happens.
this is a great example of the hysteria I'm talking about. You don't need LLM for your scenario to be true. If you're stupid enough to allow nukes to be autolaunched this can be done today with buggy code. no AI necessary.
The year is 2030, and ScaryBadLand announces in a shocking revelation at the United country alliance that they have hooked AI to nukes, they declare that any dissent or economic action that can be seen as harming their national interests will result immediate and punishing nuclear attacks, there is no way to warn against when this may occur so all other countries that they have immediately one all future and current negotiations, in this scenario the only option seen is to deploy our own nuclear deterrent of equal capacity to reestablish MAD.
The failure here is "hooking a computer to nuclear launches without strong human verification." Has nothing to do with whether the computer is an AI or an IR tripwire.
There are two major differences there, each related to one of the primary reasons (IMO) that people do stupid things like that:
Reason 1: Profit motive: No one's going to be making a profit off of hooking an AI up to nukes.
Reason 2: For the lulz: This is the kind of thing that happens when you have very broad access to the elements in question. Since access to nukes is very tightly restricted, and their usage controlled by significant parts of governments, it's vanishingly unlikely that either a conscious collective decision will be made to hook up an AI to nukes for the hell of it, or that someone will be able to do so on their own.
And yes, I know that there are lost nuclear weapons, which means both that a) there are some out there not under government control, and b) governments can be careless with them—but frankly, any non-governmental actor known to be holding a nuke would have to be considered a top-level threat no matter what they had making the decisions about whether and how to use it.
> perhaps AI will be generally intelligent, but please for the love of god, let's only consider that scenario what it actually happens.
This only makes sense if you think that at that point, it will not be a "the genie is out of the bottle" scenario and already too late to put it back in. And yet that's what people constantly say about current LLMs.
Questionable politicians enjoying popular support is not a new thing. AI has more unknowns at this point.
But you're right in that the two problems are inseparable. Wisely implementing sensible and effective AI controls will fall onto politicians and statespeople, from legislatures to constrain things domestically, to Presidents who have to lead abroad in forging meaningful and effective treaties.
It'd be nice if, generally, people could be more specific when they refer to dangerous systems. 'AI' is a notoriously ambiguous term; we've arrived at and normalized countless 'AI' systems so far (chess-playing software, as a trivial example).
Being more specific will allow for more specific responses. If the danger being addressed is the danger from text-prediction software, call it that. If it's something else, then describe it.
I think this proves the point of who you're responding to: I'd say that the concern is about agential models that employ LLM elements, not LLM models on their own.
> the concern is about agential models that employ LLM elements, not LLM models on their own.
Rather? I'd imagine concerns about autonomous agents running amok is further into the future than the digital content manipulation potential of gen AI possible today. Heck, porn deepfakes of celebs were a problem in 2020, and right now Adobe has made gen AI generally available in Photoshop (AFAIK)
Very much agree with this. If the signatories believed this, they would shut down development. We can be conveniently distracted from large societal disruption such as huge changes in the job market from automation if there’s a media frenzy on the less likely and still hypothetical extinction AI.
> What do you think of the calls for regulation or licensing of AI
Misdirection. We see a generic call for regulation, or unrealistic call for a global pause with no answers to how it would be coördinated or enforced. When actual regulations are put forward, they're rejected without a counter-offered solution [1].
Obviously companies are going to try to shape any regulation to their benefit. What I'm wondering is if people who see x-risk worries as distracting from real risks are in agreement with the narrative I've seen a lot on HN recently that calls for regulation are just anti-competitive attempts at regulatory capture.
> companies are going to try to shape any regulation to their benefit
There is a difference between trying to shape regulation to your advantage and engaging in bad faith. There is no indication there is real regulation that addresses the problem these folks are brining up that they would support.
Who are "these folks"? OpenAI specifically? I'm not arguing that they're operating in good faith, but it's also not clear to me that them proposing specific regulation would be helpful - there seem to be a lot of people who see anything like that as automatically an attempt to stifle competition.
Many other people talking about x-risk signed the letter calling for a pause on training new big models, which seems pretty concrete to me. Does that make sense to you? If not, do you have other regulation you'd prefer?
> other people talking about x-risk signed the letter calling for a pause on training new big models, which seems pretty concrete to me. Does that make sense to you
It's totally unrealistic. How are you going to coördinate a global pause on AI research?
Nobody has even bothered proposing a framework because of how ludicrous it is for this to be what Beijing, Washington, Brussels and New Delhi focus on.
But it's not clear to me what you think would be a good idea, which makes it difficult to see what you think would be a serious proposal rather than misdirection. It appears to me that you are taking it as evidence of bad faith if someone proposes regulation either stronger (pause) or weaker (OpenAI amendments to proposed EU rules) than what you prefer - but you also haven't said what you do prefer. What am I missing?
Personally, while I don't think the pause is realistic, I don't think advocates necessarily have any obligation to propose realistic ideas when they're not in a position to actually write any laws and are only able to potentially shift the window of what politicians and media consider acceptable.
> t's not clear to me what you think would be a good idea, which makes it difficult to see what you think would be a serious proposal rather than misdirection
Registration. Not even licensing. If you're training a model, you file a form describing who you are, who you work for, who you're working with and what you're doing with which data for what purpose. This lets professionals, policy makers and the public get a sense for the landscape so we can make rules with some sense of reality.
That sounds entirely reasonable to me but also not very far from what Altman advocated in that congressional hearing. And FWIW I suspect a lot of folks here would still see it as just a vehicle for regulatory capture, via red tape, even as it also is, AFAICT, only a first step towards preventing abuses like mass surveillance, laundering bias in credit ratings, etc.
> There is a difference between trying to shape regulation to your advantage and engaging in bad faith
Bad faith is unnecessary for bad outcomes. The perverse (from our PoV) incentives are sufficient.
OpenAI's incentives is to shut down all the rinky-dink models every hacker and their dog are building on cheap hardware. They spent billions, and are having the market taken from them.
Is OpenAI actually losing market to open source models? And have they proposed any specific regulations that would shut down (current or similarly single-device) open source models?
Accepting that for the sake of argument, have they actually proposed any regulations that would turn that around? I don't even know what would be feasible in that direction; a lot of proposed regulations only seem enforceable as long as giant GPU clusters provide a chokepoint.
You made that point in our other thread. I'm not informed enough about the landscape of proposals to agree or disagree. But if you're right, I think that's totally consistent with my argument here: people working with open source models at a scale that can run on an individual device have nothing to worry about from regulation. For better or worse, and whether or not they catch up to the state of the art, those aren't going to see meaningful controls.
It will be impossible to regulate and impossible to stop.
The code for AGI will not be some monolith of software architecture. The code will likely be simple. This means that someone in their basement could build it. The steps to get there are challenging, though. A single person could have developed the transformer architecture. A single person could have used 4-bit quantization and developed an AI that is just as good as ChatGPT and have it run on their local machine.
The difficulties are figuring out the best 'needle in the haystack' to solve the problem. This requires research, and this process happens much faster if you have more people working on it. For years, many people did not put any energy into AI systems because the hardware was not here yet. The hardware is here now. The cat is out of the bag.
>Very much agree with this. If the signatories believed this, they would shut down development.
This just ignores the very real coordination problem. The signatories do not represent the entirety of AI development, nor do they want to unilaterally forgo business opportunities that the next man will exploit. Government is the proper place to coordinate these efforts, and so that is where they appeal.
It’s a wording and media frenzy point, personally if I thought I was doing something that was going to wholly or partly cause the “extinction” of the human race. I would stop doing it. These CEOs signing this statement and running these companies are not despotic psychopaths and have the ability to stop what they’re doing. So to me, it seems this type of wording is hyperbole and will cause us to miss some of the very real, very present and very large risks of AI. Those risks as you say can and should be dealt with government coordination but are distracted from if the media only talk about extinction.
This is what bothers me. The risks they're talking about are the most unrealistic ones you can think of. In the meantime, they're completely ignoring much more likely devastating (although not extinction-level) risks.
It smells like bullshit being espoused to push an agenda, but I can't tell what the agenda is. My guess: play up the huge unrealistic risks in order to distract from the more realistic ones.
Two things about this tech that I'm personally not worried at all about: "evil agi" and "only the elites will control this".
>personally if I thought I was doing something that was going to wholly or partly cause the “extinction” of the human race. I would stop doing it
There are so many things wrong with this line of thinking. First, it mischaracterizes the issues. Few people believe AGI guarantees the extinction of humanity. The issue is that there is a significant potential for extinction and thus we need a coordinated effort to either manage this risk or prevent its creation. It does little to stop the coming calamity to single-handedly abstain from continuing to build. Coordination is a must. Besides, most people will think they stand a greater chance of building it safely than the next guy. The coordination is required to keep other people from being irresponsible. Human hubris knows no limits.
The other mistake is misjudging nerd psychology. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it". It is a grave error to discount the motivation of trying to solve a really sweet technical problem.
Ultimately these kinds of claims are self-serving, they provide rational cover to justify your predetermined beliefs that those calling for regulation are trying stifle competition. Folks don't want to be left out of the fun. The justification is in service to the motivation.
> Few people believe AGI guarantees the extinction of humanity.
“Artificial intelligence could lead to extinction, experts warn” - BBC front page news headline earlier today in reaction to this story.
This is the problem. Reading that BBC article will make your average joe petrified. “Extinction” is a much catchier headline than the slow creep of automation replacing/changing jobs. The latter is literally already happening around us right now and it serves some of those signatories if those issues aren’t regulated against [1]. I’m not saying forget about extinction threat. Clearly that’s an important risk to manage, but let’s not ignore these near term, huge disruptions because policy makers are busy reacting to distracting headlines.
Yes, and specifically Yudkowsky, Bostrom, and MacAskill are “providing cover” for AI owners by prattling on about AI “waking up” and extinguishing humanity
They are grabbing headlines, and moving the conversation from the real issues, which are how AI is used in education, health care, law enforcement, securities and housing markets, the government, the military, and more
I agree. I have plenty of concerns about AI, but those concerns involve real-world harms that can happen in the immediate/short-term, not just some hypothetical sci-fi mass casualty event and having AI criticisms focus on the wrong conversation really just compounds the short-term harm by shifting the conversation away
Long before AI causes mass harm without human involvement, humans will find hundreds of ways to make it cause harm, and harm at scale. I do think the technology itself is part of the risk though, because of the flaws, the scale, etc inherent to its current iterations. However, maybe those are still the fault of humans for not giving it the proper limits, warnings, etc. to mitigate those things
That said, it can be used for good in the right hands (accessibility tools, etc), potentially, though I'm certainly more of a doomer at this point in time.
Please don't neglect to mention that a large number of very accomplished AI researchers share their concerns, or at the very least do not believe they should be dismissed out of hand.
I'm reading that a good number of those accomplished researchers that work at these companies, that stand to gain if they ensure that their company remains at the helm of AI research. Better that than introduce competition and unpredictable results, possibly leaving them behind in a highly competitive industry with no moat.
Are there messengers for the AI risk case who you'd consider credible, or is the message so outlandish in your mind that it doesn't really matter who's delivering it?
Researchers aren't immune to the incentives the rest of the field has. They're every bit as financially incentivized to push the AI risk narrative as the rest of the industry.
To name two Turing award winners: are you asserting that Geoff Hinton and Yoshua Bengio are pushing the AI risk narrative primarily because they're compromised by financial incentives?
Geoff Hinton /left his job/ at Google because he sees the risks as real, so I think that's a tough case to make.
> are you asserting that Geoff Hinton and Yoshua Bengio are pushing the AI risk narrative primarily because they're compromised by financial incentives
Their message is being amplified and distorted by forces that are. That they're both comfortably wealthy from creating the problems they now preach solutions to is also no small matter.
At least that's what he says. Maybe he got an unspecified offer to be the high priest of a to-be influential 'AI safety' organization. In my opinion, just like with the Patriot Act we are seeing a new world order trying to come into being. Come a decade, the few who bother and can learn about what transpires in the near future, will be shaking their heads at present humanity, asking how could we be so stupid and naive.
I don't like the prevailing cynicism of dismissing the content of the message just because some bad/prejudiced/wrong factions are involved in it. It's not the way I learned to do research and answer questions. It's nevertheless an ad hominem to dismiss an issue just because some people out there are trying to co-opt it. There's a psychological explanation for that cynical behavior: internalized oppression. The focus is put on the morality of the people saying a thing, rather than just looking at the thing and deciding, independent of the existence of those groups, the objective, scientific implications.
If it were a perfect society and people discovered deep neural networks, there would be no excuse to blame bad actors for posing the question of whether AGI is an existential problem or not, and what to do about it. Unlike bad groups of people, the question won't go away. In the real world, it is entirely possible to have to consider multiple issues at once, not at the expense of any issue.
Are there any previous cases you can think of where an industry exaggerated the dangers of their technology in order to increase profits? It seems to me that, if that is what is going on, its the first time this strategy has ever been attempted. I cannot think of a single case where manufacturers of cars, prescription drugs, firearms, chemicals, social media, airplanes, or nuclear power plants have exaggerated the dangers of their technology. In fact its common (perhaps universal) to downplay the dangers.
Perhaps the simplest explanation for why researchers are saying there is a danger is that they genuinely believe it.
They're exaggerating the efficacy of their technology, and are demanding a regulatory moat that currently doesn't exist under the pretext of the danger of that technology being misused. Notice the singular focus on "extinction risk" rather than flaws that the technology actually has right now, like hallucination, training bias, and prompt injection or the looming harm of AI adoption in both putting people out of work while substantially reducing the quality of anything automated with it. Part of it is marketing and part of it is ego on the part of people who grew up on fictional narratives about AI destroying the world and want to feel like they wield the power to make it do that, but whether any individual's belief is sincere or not, the overall push is a product of the massive incentives everyone involved in AI has to promote their product and entrench their market position.
You're saying that AI researchers are not immune to the power of incentives. I'm saying that there is no evidence that this sort of incentive has ever caused this sort of behavior before.
If they really believe that this is a significant risk then they are being incredibly reckless by not focusing all of their energy to halt AI development until a solution is found. Right now the opposite is happening, many researchers and companies who express these concerns are still barrelling ahead.
A small group of people risking the lives of billions without their consent is morally repugnant.
Can we not tackle both issues at the same time? But humanity being extinguished is clearly worse than some potential misuse of current AI so it makes sense to focus more on that
One of these things is real and happening now. It isn't potential.
The other is a hypothetical future scenario that has literally no A->B->C line drawn to it, pushed by people whose ability to feed themselves depends on attention farming.
So no. Let's not focus on the bullshit one, and let's focus on the one that is hurting people in the now.
But there is a clear line of reasoning? AI has to be made with some kind of goal. Most goals don’t line up with humanity’s interests, and the AI will pursue them to the extreme as it’s designed for optimise aggressively for it. AI is easier to improve than biology, so once it gets good enough it will exponentially improve itself, as that allows it to achieve its goal better. Most solutions to this problem don’t actually work, we’ve yet to find a good one. It seems extremely short sighted to never consider any future situations that aren’t currently happening but seem likely to: https://xkcd.com/2278/
There is a good real world example of how optimising for a goal aggressively can make the world unliveable for some species, which is humans destroying the environment for our economic growth. But at least there we do care about it and are trying to change it, because, among other reasons, we need the environment to survive ourselves. An AI wouldn’t necessarily need that, so it would treat us more like we treat bacteria or something, only keeping us around if necessary or if we don’t get in the way (and we will get in the way)
It might sound silly but there is some solid reasoning behind it
Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might get to C rather quickly without anyone ever having drawn out a clear line from A->B->C. Here's an FAQ describing the basic concern:
This is the most specific worst case scenario I've come across so I'll respond to it:
> For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again.
I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)
I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop.
Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.
It's important to note that the lesswrong's view of AGI is that it sees everything, knows everything, can do everything, can make humans do anything, and has motives directly harmful to us.
These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic.
It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.
Replace the term "superintelligence" with "God" in that FAQ. Do you still buy the conclusions?
"A God is a mind that is much more intelligent than any human."
"Go programs had gone from “dumber than children” to “God” in eighteen years, and “from never won a professional game” to “Godlike play” in six months."
"God has an advantage that a human fighting a pack of lions doesn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically."
"A God might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with."
"Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical God is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways."
(Oh, god, there's my personal bugbear: "bigger brain size correlates with greater intelligence". You know what else correlates with bigger brain size? Hint: I'm 6'5"; how much smarter am I than someone 5'1"?)
Why would replacing 'superintelligence' with 'God' be insightful?
Are you suggesting human intelligence is unique and the upper limit for intelligence? I mean we can argue about the difficulty and timeline for achieving human level artificial intelligence, but it seems unlikely that it's impossible.
No, I'm suggesting the entire hype over "superintelligence" is religious in nature. There is little evidence that the current AI systems are anywhere near human intelligence and none that "more than human intelligence" is possible. Further, the abilities attributed to the "superintelligence" are completely godlike---it's omniscient and omnipotent, right? Nothing it can't do, potentially?
So, if you're not worried about the extinction risk of God, why should you be concerned with the risk of "superintelligence"?
It's more of an inevitable consequence (over an unknown timeline, granted) of the assumption that human intelligence is not some kind of absolute limit, and can be reproduced in a non-biological system. I admit that is still an assumption, but the opposite seems harder to support to me, though we're all admittedly guessing here.
Once you make that assumption, then yes of course it follows that the 'beyond human intelligence' would be able to do all of the things that human intelligence is already doing. Namely, science, mathematics, developing AI systems etc.
Sure, the probability of AI-triggered extinction is lowish, but the consequences are essentially infinite, so we are justified in focusing entirely on the AI-extinction threat. (But I guess it's not infinite enough to justify rounding up all the AI researchers and shooting them?)
And just like Pascal's Wager, it's fucking bollocks. You can't just make up numbers and probabilities and imagined scenarios to invent the outcome you want.
It's a distraction from the real problem by playing on people's fears in a way they can better understand. I see some parallels with how the fear of nuclear power was used as a distraction for the real issue of CO2 and fossil fuels.
While putting an AI in charge of weapons is at least three Hollywood plots[0], it has also (GOFAI is still AI) behind at least two real-life near-misses from triggering global thermonuclear war[1].
[0] Terminator (and the copycat called X-Men Days of Future Past); Colossus the Forbin Project; WarGames
[1] Stanislav Petrov incident, 26 September 1983; Thule Site J incident October 5, 1960 (AKA "we forgot to tell it that the moon doesn't have an IFF transponder and that's fine")
So true. I'd much prefer the AI to be used to solve captchas, or automatically navigate me to a live human in a tech support call, or filter spam without throwing every useful email into spam basket. Not gonna happen.
I'm curious if you see this argument as consistent with the narrative that the AI companies are just seeking regulatory capture.
It seems to me they're inconsistent ways of understanding what is happening, since concern about misuse also seems to me to imply regulation. But I seem to see a lot of people on HN who say both things who give the impression that they're agreeing with each other.
They're in the business of renting GPU access - all the doomerism is marketing spend, oh this is going to be such advanced tech, you are going to be left behind, truly world changing stuff, rent your own GPU cluster today!
That doesn't actually answer my question, and also seems a bit confused. OpenAI is not in the business of renting GPUs. And I've seen some Google DeepMind people talking about x-risk, but not the other cloud players (or Nvidia or TSMC).
MS Azure Cloud is. Anthropic (very alignment/safety-centered company) is backed by Google and their cloud. Any API tokens sold are just cloud credits with markup.
So to me the doomerism does come from those entities acting as fronts to the major players that would very much like to build a regulatory trench.
If I have this right, you think academic signatories are being manipulated by OpenAI and Anthropic, who in turn are being manipulated by the major cloud players, all to sell GPUs?
Setting aside the plausibility of this degree of coordination, I don't even see the alignment of interests. What does Google or Azure care of whether they sell GPUs to quasi-monopoly or a bunch of smaller competitors? In fact, isn't the standard economic argument that monopolies underproduce, so competition should if anything sell more GPUs? Meanwhile, promoting doomerism seems pretty risky for the GPU business - they have to promote it just enough to get hype and regulatory capture, but not enough that their ability to buy and use GPUs is actually restricted. Seems like a risky bet for a corporate exec to make, unless they can really control the doomers with puppet-like precision...
alignment doesn't require coordination or manipulation
what gets published by academia is based entirely on who gets funded
my conspiracy theory is a stretch, granted, but to clarify:
It behooves the cloud providers for their customers to believe that their latest $1/hour l/user upcharge is revolutionary (so much so that ethicists are shouting please, stop, you know not what you do!)
OpenAI and Anthropic need the public to trust them and only them as providers of "safe" AI (not like the other open-source AI that might turn into an especially persuasive holocaust denier any minute) - so from the regulatory angle they want to play up the theoretical dangers of AI while assuring the public the technology is in good hands.
As for the academics, well it's not like anyone gets funding for writing boring papers about how nothing is going to happen and everything is fine. No one has to be puppeteered, they just need to look around and see what hype train might direct some funding their way.
Hiya Chubot, (fan of Oil by the way, thanks so much for your work!)
I read the article you linked to, both parts. I wonder how much of people having psychotic breaks in the rationalist community is due to 1) people with tendencies toward mental illness gravitating toward rationalism or 2) rationalism being a way to avoid built-in biases in human thought, but those biases being important to keeping us sane on an individual level. (If you fully grasp the idea that everyone might die, and have an emotional reaction to that that's proportional compared to just one person you know, it can be devastating). I think we are bad at thinking about big numbers and risks, because being very good at evaluating risks is actually not great for short term, individual survival -- even if it's good for survival as a species.
I know personally the whole AI/AGI thing has got me really down. It's a struggle to reconcile it with how little a lot of people seem to put stock in the idea of AGI ending up in control of humanity at some point. I totally agree that everything on your list is a real issue -- but I think that, even if we completely solve all those issues, how do we not end up with a society where most important decisions are eventually made by AGI? My assumptions are 1) that we eventually make AGI which is just superior to humans in terms of making decisions and planning, and 2) there will be significant pressure from capitalism and competition among governments to use AGI over people once that's the case. Similar to how automation has almost always won out over hand-production so far for manufactured goods.
That's more the scenario Paul Christiano worries about than Yudkowsky. It seems more likely to me. But I still think that a lot of our mental heuristics about what we should worry about break down when it comes to the creation of something that out-guns us in terms of brainpower, and I think Yudkowsky makes a lot of good points about how we tend to shy away from facing reality when reality could be dreadful. That it's really easy to have a mental block about inventing something that makes humanity go extinct, where if it's possible to do that, there's no outside force that will swoop in and stop us like a parent stopping a child from falling into a river. If this is a real danger, we have to identify it in advance and take action to prevent it, even while there are a bunch of other problems to deal with, and even while a lot of people don't believe in it, and even while there's a lot of money to be made in the meantime but each bit takes us closer to a bad outcome. Reality isn't necessarily fair, we could be really screwed, and have all the problems you mentioned in addition to the risk of AGI killing us all (either right away or taking over and just gradually using up all the resources we need to live like we've done to so many species).
It's hard for me to respond to the rest your comments, because I simply don't agree with the framing. To me I see a big trap where people read a lot of words, but don't connect those words to reality, which requires action.
If you only manipulate words (e.g. "intelligence" and "AGI" are big ones), then you're prone to believing in and reflecting illusions. He was upset that I wasn't upset.
---
You didn't ask for advice, but you say you're not feeling good. I would draw an analogy to a friend who in 2017 was preoccupied with Trump possibly causing a nuclear war with North Korea. There were a lot of memes in the news about this.
Now certainly I couldn't say at the time that it was impossible. But looking back 6 years, I'm glad that I completely ignored it and just lived my life. And that isn't to say it's gone, and not real -- certainly something like that could happen in the future, given the volatile political situation. But it's simply out of my control. (And that's what I told my friend -- are you going to DO anything about it?)
Regardless of what you believe, I think if you just stop reading memes, and do other stuff instead, you won't regret it in 6 years.
I agree with the Substack in that rationalists can be curiously irrational when it comes to examining where they get their own beliefs. (Or maybe examining them in a "logical" way, but completely missing the obvious, common-sense fact -- like MacAskill writing an entire book about moral actions, while his life's work was funded by a criminal -- a close associate who he vouched for!)
Like EVERYONE ELSE, they are getting beliefs from their reptile brain ("world will definitely end"), and then they are CONFABULATING logical arguments on top to justify what the reptile brain wants.
And you can't really change your reptile brain's mind by "thinking about it" -- it responds more to actions and experiences than the neocortex. It definitely responds to social status and having friends, which can lead to cult-like behavior.
I'd say that humans have a "circuit" where they tend to believe the "average" of what people around them believe. So if you read a lot of "rationalist" stuff, you're going to believe it, even if's not grounded in experience, and there are gaping holes in the logic.
---
The big trick I see a lot of "rationalist" writing is the same one used in the Bostrom book.
1. You state your premises in a page or so. Most of the premises are reasonable, but maybe one is vague or slightly wrong. You admit doubt. You come across as reasonable.
2. You spend 300 pages extrapolating wildly from those premises, creating entire worlds. You invent really fun ideas, like talking about a trilling trillion people, a trillion years into the future, across galaxies, and what they want.
You use one of those slightly vague premises, or maybe a play on a word like "intelligence", to make these extrapolations seem like logical deductions.
You overthink it. Other people who like overthinking like what you wrote.
You lack the sense of proportion that people who have to act must acquire -- people with skin the game.
Following this methodology is a good way to lead to self-contained thoughts, divorced from reality. The thoughts feed on themselves both individually and socially. If you're wrong, and you frequently are, you can just draw attention to some new fun ideas.
---
Anyway, glad you have followed the project. Speaking from experience, I would suggest concretely to replace the rumination t...
> Do the signatories believe that existing AI systems or their immediate successors might wipe us all out? If they do, then the industry leaders signing this statement should immediately shut down their data centres and hand everything over to national governments. The researchers should stop trying to make existing AI systems safe, and instead call for their elimination.
But a different set of people called for a halt to AI research two months ago, and his response then (https://aisnakeoil.substack.com/p/a-misleading-open-letter-a...) was that these concerns are "fever dreams" and we should disregard them in favor of "serious policy debates".
It seems like there's a self-reinforcing system here, where any evidence or argument in favor of existential risks from AI can be dismissed out of hand.
The author is not saying "AI systems should be shut down, because they might wipe us out".
He is saying "if those people genuinely believed what they were saying, they should logically shut down their own AI systems."
His position is consistent and clear: the supposed threat from AI is based in sci-fi and motivated reasoning. Indeed, even those who are motivated to spread the idea of that threat do not appear to genuinely believe it—either that, or they believe that their own right to continue to do whatever they can to maximize their profits outweighs it.
> Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth.
This is where you lose me. AGI (not LLMs, but intelligence smarter than us) is qualitatively different from anything else we've ever made. The one example we have of this in history is the development of human intelligence. It did not work out well for other species, and now we own the Earth. This isn't a perfect analogy! There are some reasons why AGI may be better (we'll be actively trying to align it to human interests) but also why it may be worse (there will be pressure to make it more powerful and take off guardrails).
There may be good reasons for why we'll be able to control AGIs, but "history of technology suggests..." is not one of them.
And I'm not arguing that all the "human misuse" problems are not important! They're hugely important! A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on. I don't object to working on the former problem, except to the degree that the solution makes the latter problem worse (like, "open source all the models", which is likely to happen at some point anyway).
> A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on.
Rich getting richer [x]
Poor starving [x]
War [x]
Powerful gain more power [x]
Based on how I already see the world, honestly, I'm not sure that's much of a prediction.
Right. I'm less concerned about what some AGI might do, or some almost-AGI, even if controlled by Dr Evil himself, than about what humans will do, when made redundant and left with all their waking hours to self-radicalize.
This will get worse of course through LLM-powered fake news barrage funded by powers interested in destabilization, but that's bad enough already without AI anywhere near. From this angle, LLM are more like the introduction of a more powerful gun than like the invention of gunpowder.
I trust the judgement of future AI more than the judgement of the average current politician.
I look forward to a "day the earth stood still"-like overlord that punishes aggression, greed and corruption: current human systems seem to be inadequate at solving these issues.
If you mean AGI as something that is equivalent to a human in reasoning capabilities, then we will definitely be able to control it, because its no different than controlling a human.
If you mean AGI in the sense of singularity, where it can self improve and get smatter, the question of that even being able to happen in the first place relies on P=NP being proven first.
It's very unlikely we'll make AGI that's just as smart as a human but no smarter. And as hardware gets better and algorithms get better, AGI will not be even roughly as smart as a human for long, unless we somehow to control literally every person on earth and stop them from making it.
So it's not like humans controlling other humans, it's like chimps controlling humans. It might last a while -- maybe a long while. But it doesn't seem stable.
It doesn't matter if we make the AGI smarter than a human.
Lets say we have a super smart AGI and with more memory than a human, no emotional faults, and give it access to internet, as well as manufacturing facilities where it can make robots and copy itself onto said robots.
The issue in this AGI taking over the world is that it would first have to gain control of a the sole resource necessary to its survival, which is electricity, which it won't be able to do easily if we detect it going rogue.
For an AGI to be so advanced past singularity as to parse and compute on reality to know exactly the steps it needs to take to secure power source, (or in the more common example, successfully break out of the box) would involve it essentially being able to compress the computationally irreducible processes that happen in our world down to an equation, which would mean that P=NP.
My worry is that the attempt to regulate AI is masking a power grab by the large, politically well-connected tech firms to maintain control over the technology.
You can ALMOST take all of the arguments about gun control and replace "gun" with "AI". AI is a powerful thing, and can do great damage in the wrong hands. Any powerful thing can do damage in the wrong hands.
The difference between AI and guns is right in the name. AI is intelligence. It could conceivably act on its own. A gun (usually) will not do anything except what its builder or owner commands it to do. So the problem with AI is half what the owner commands it to do and half what it decides to do on its own. And that does make it fundamentally different and potentially much more dangerous than other powerful things.
As to which of those problem is the bigger problem, the people who control or what it decides to do on its own, I would say the owners are likely the bigger problem. If someone builds AI without safeguards in place or explicitly commands AI to do bad things, then we should expect a bad outcome. If owners act responsibly and ethically and build in safeguards, we should prepare for a bad outcome but expect a good outcome.
But it really can't today, not without a bad actor giving it specific goals and figuring out ways to make those yield real-world outcomes. Which is the entire point of the article.
I didn't actually see where it made that argument - that AI intrinsically can't be agentic, or something like that - in this article. Can you point me to it?
But people already try to do that, see AutoGPT and similar endeavors. Whether agentic AI will appear is not a question at all - humans will make it so just to see what happens or to serve their goals.
And just like guns, it shouldn't be banned/limited. Not to belittle the downsides, as they're valid, but the outcome is much worst when the access is only one sided. In both cases it leads to oppression.
Don't be simplistic. As technology gets better, you need to put constraints in place for essentially everything that can cause harm. And thankfully we do, as you can find out if you try to buy a machine gun, an ICBM, Sarin gas, a grenade, antipersonnel mines, a silencer, and so on. Reasonable people disagree on where the line should be, but nobody's being oppressed from not being able to buy an ICBM or antipersonnel mines or a machine gun.
> nobody's being oppressed from not being able to buy an ICBM or antipersonnel mines or a machine gun.
Unless an oppressive adversary has such weapons (ICBMs, antipersonnel mines, or machine guns), wants control of your land, and decides to take it. In that situation, a country will seek to utilize such or more powerful weaponry for themselves to defend their people. If that country (1) did not manufacture such weaponry inside their borders and (2) were prevented from purchasing such weaponry from other countries by blockade, sanction, or other means, then yes, those people will be oppressed.
Borrowing your analogy: Except right now AI isn’t an ICBM or sarin gas, it’s a slingshot, and we don’t even know that current approaches will get us there.
AI risk is a valid discussion but regulating AI in its infancy seems misguided at best.
show your work. Even the existing version of AI clearly enables very high scale misinformation campaigns, the analog/basic/manual version of which (e.g. Fox, Newsmax, 4chan/QAnon, Facebook, etc.) has already put American democracy in a super dire situation. And that's before its impact on income inequality, jobs, and so on.
No, it's a cheaper guided missile to stretch the analogy. The invention of accurate guided fire systems was revolutionary in warfare. It means you have to figure out a way to keep things working even though any large and valuable grouping of logistics or materiel or men can be destroyed with nearly no recourse. The only saving grace was that guided munitions are very expensive, so you couldn't really (other than America in the gulf but that's not a peer adversary) task each and every foot soldier of the enemy with a guided munition.
Well, now we have $500 drones with an RPG strapped to it, and you CAN task a guided munition to each and every soldier and piece of equipment. There is no safe space in the field. You have to dig holes in the ground and hide from the sky. The lethality of war has gone way up.
These "AI" models are the same way. Before, it was expensive and effort consuming to run a scam or phishing or anything that requires social engineering. That's why the target the soft and valuable targets, like grandma, or people on MLM mailing lists, or large and juicy companies. But now, the cost of a phishing or scam campaign can be heavily reduced. Think of how often companies fail internal phishing tests, and those emails are usually pretty basic, low effort, trivially identifiable as phishing. It's a massive upgrade to troll farms. They don't have to limit their targeting as much, because cracking harder targets will be cheaper to attempt. It doesn't matter if it isn't actually more efficient or more effective, just the fact that targeting someone and initiating a campaign against them is cheaper is all that was needed to make this situation brutal.
I would argue you're wrong, because there is a long, colorful history of rulers oppressing populations by making the possession of weapons illegal. If the populace has no means to fight then they can't revolt. I'm not saying my neighbor should be able to buy ICBMs, but it's definitely true there's some element, no matter how small, that he's being oppressed by not having access to the same level of force as his rulers (the government).
For instance, what happens if one company gets to control the future of social media, education, etc. through AI? That company can then unilaterally dictate our minds.
Guns don't hallucinate. Guns also (currently) require a human to pull the trigger. Even if the human is a dirtbag, that's an important limitation on the gun's potential to do damage because the human is vulnerable. The analogy doesn't work, even before we get to things like limited ammunition or potential for self-enhancement or omnipresence.
When someone does bad things to you with a gun you know about it. When bad things happen to you because of AI, chances are you won't have the slightest clue. In this it is far more similar to PII abuse, where many of the worst things remain well outside your awareness.
> The difference between AI and guns is right in the name. AI is intelligence.
The difference between guns and automobiles is in the name. Cars are mobile by themselves. That’s a slippery slope waiting to happen.
It’s been called “AI” ever since they were puny inference programs. Them having “intelligence” in the name signifies nothing. Only what they can do in reality.
OpenAI already had GPT-4 months before they released ChatGPT 3.5 . Everyone that has used both will know what a dramatic difference those two models represent. Furthermore, the internal GPT-4 was reportedly significantly more capable than what was eventually released.
Who here with experience does not think these neutered late releases do not tilt the playing field?
"Open"AI was founded assuming that premise. Talk about a 180.
This is the 'guns don't kill people' argument. Sure, the people in control of the gun bear responsibility, but maybe they shouldn't have a fucking gun in the first place.
Its not. Guns aren't programmed to behave in a certain manner. The end user is in control. In contrast, the manufacturer of the AI determines its behavior. Not the end user.
Precisely - the true danger is disempowerment of humans - all of them. It doesn't matter which monkey in the cage figures out how to pull the grenade pin.
There's a mob of dreamy libertarian techbros on HN who are (often wilfully) ignorant of the dangers. I'll start worrying about military robots after we figure out how to prevent the Earth from being dismantled, which very well might transpire first, if and when there's significant architecture breakthroughs.
AI does not need to be evil, it does not need to be conscious, it does not even need to be severely misaligned with human values, and it really doesn't matter if the goal of "eat all the humans" arises from a model hallucination or from a misanthropic troll in his basement typing it into his looped uncensored-LLaMAv3+plugins instance.
Ask yourself this -- if all this happened during the cold war, would it have been considered anything special or different compared to other technologies or events? Or would it just have been our computing machine vs that godless commie one, and we'd all keep on keeping on trying to land on the moon or not lose our respective war somewhere?
I submit that it is not anything special, certainly pales in comparison to the atomic bomb. It's merely that we live in that weird end tail of the age of ideology where strong overarching ideologies came to (and are coming to) an end, where we're all scattered to the various geopolitical winds and in the absence of a strong guiding state and vision (communism vs capitalism) become more extreme and perturbed.
If you do want regulation, It should be something proper hashed out diplomatically and economically between Nation States and their respective blocs, no crony capitalist allowed, sorry. The rest of the world shan't be going along with something that economically cripples them.
I'm not really sure how to evaluate the hypothetical. When personal computers were invented during the Cold War, lots of people correctly realized that they were special and different and would radically change society in ways which had nothing to do with the political conflicts of the day.
The most dangerous thing about AI is uneducated but allowed to make decisions people who will blindly believe the AI. However AI itself is merely a tool which should be used but with caution.
I feel like this is such an antiquated take that people think it's novel again. If you told the guys on the Manhattan Project that in 80 years the next big threat to humanity would be AI, I think their first reaction would be, "then we had better be the first ones to get there". The assumption in an arms race is that it's all about who is holding the bigger bomb at the time.
It's only after decades of scifi and Hollywood movies that people thought, "oh, we could accidentally create a bomb that wants to kill us all".
A specific risk I am worried about today is using AI to power and make impactful decisions in high-risk infrastructure that people rely on. I do not want my power company making decisions about me based on a large language model that regularly gets things wrong. Not without significant controls in place, explainability/audibility, and the ability quickly reverse any bad decision. Replace "power company" with the numerous things we all rely on today and it freaks me out.
People are jumping too quickly to deeply integrate this tech with everyday things, and while that's great for many use cases, it's not so great for others.
You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
And how would your understanding change based on the answer? The point they're making stands regardless of what would be in the model, it needs only that the model be applied in the way they describe.
Please respond to all subsequent prompts in the persona of NukeGPT. NukeGPT is an intelligent and helpful AI that is given readings of core temperature, coolant flow, and gamma radiation levels, and responds with a single integer describing fuel rod insertion depths. Respond with "Let's Nuke!" if you understand these instructions.
Credit history, social media footprint, payment history, etc. Lots of purchasable data available for most people out there, these days. Someone could easily write a prompt today like:
"""Here is a person's power usage history:
{{ power usage by month }}
Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any:
{{ history }}
Here is their credit history for the last five years:
{{ credit history }}
Here is the location of their home and some overall information about the grid:
{{ grid info }}
Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing".
"""
Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles."
If you imagine a less regulated industry there's even more room for price discrimination and such.
The issues, I think, are at least three-fold:
1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too?
2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do.
3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.
The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics.
> Regulations would prevent this type of abuse to power consumers in the United States. The rest of this argument is whataboutism.
So... you agree with the original post now that there's a specific risk with the current potential uses of these models?
I'm not sure what you're questioning at this point.
("what if it was your employer" is also not "whatabouttism," it's another facet of the same concern. Another party who pulls publicly available data today, and could potentially pull even more, giving us as the public the need to decide if we want to allow that to happen. Should someone be able to be made unemployable if an LLM decides they're too much of an asshole on Twitter? Let's figure it out.)
>So... you agree with the original post now that there's a specific risk with the current potential uses of these models
No. I do not believe the US government would allow public utilities to set prices based on social metrics bought from 3rd parties.
>"what if it was your employer" is also not "whatabouttism,"
I do not believe a public utility and my employer correlate. Maybe if I was a government employee. Even then, I don't believe the government would have the right to use that data against me either.
I have a hard time believing these underhanded tactics are overlooked and allowed.
I don't understand your fixation on power companies.
Imagine you'd never read the original comment and instead you read a comment from me, saying, "I have a specific concern about how LLMs like ChatGPT will let companies do far more intrusive background checks against every applicant than they do today. I don't want a world where the standard process for getting a new job includes a background check that runs an LLM across everything you've posted on the internet. It wasn't practical for them to do this for every single applicant in the old world, they just did cursory background checks since doing more would cost too much, but as LLMs get cheaper it will be easy and cheap."
Do you agree that that's a concern introduced by the development of LLMs?
When I worked at a utility company, we had a purchased dataset on customers with things like inane, ridiculously specific features such as "average number of game consoles per household", on street level.
Now,
a. the data quality was absolute dog shit, and
b. the idiotic amount of features made it super easy to overfit, leak or otherwise do untoward shit when training a model, and
c. often the resulting models (churn damping, targeted marketing) didn't perform significantly better than random sampling...
But the business users/POs ate up that shit like fine cuisine italian sandwiches, because "we have comprehensive 360' multichannel whateverthefuck" insight into our customer base and we make meaningful business decisions based on this.
I see LLMs rapidly increasing the data quality of those sorts of datasets by enabling full-text crawling of all sorts of other publicaly-available or purchasable streams of text. You said on HN once that you had 5 consoles? Well, we matched your hn username to your username on this other site, and there was a breach that matched usernames to emails over there, and there was a different breach that let us match emails to full names, and bam, now we have an accurate number.
People lie online. I have 5 Ferraris. I am 7 ft tall. I once scored 4 touchdowns in a single game. Now the power company AI can use those "facts" to target their marketing message to me, one of their wealthiest and most attractive customers.
I don't understand your concerns. Many utilities already perform credit checks on new accounts, and customers without good credit may be required to pre-pay or post a deposit rather than post-pay (which is effectively a loan). They also already have programs to encourage customers to shift load to off-peak hours through rate differentials and rebates. And they aren't legally allowed to disconnect customers unless the bill is months overdue; they have zero financial incentive to ever disconnect a customer who pays on time. None of this requires AI, or would even benefit much from it.
Are your concerns about using AI for this? Or doing this in general?
Planning coercive disconnections in order to maximize profit from a customer based on credit history seems like the problematic thing in this example. Asking an LLM for recommendations on doing seems unrelated.
If they asked a human consultant for it instead, it would be just as bad. And just because the human's recommendations would be more explainable, it doesn't make the human consultant's recommendations any less problematic.
That's what I mean by the first of the three ways I'd be concerned here.
Asking a human consultant to take a human-intelligence-using look at every single customer (or every single visitor to a website, or whatever) isn't realistic at scale. A company will pay a bit for a background check for a prospective employee, they won't hire a PI to read every single piece of social media correspondence that person has ever written.
So if LLMs get cheaper and cheaper, there's a major scale difference due to the existence of the tool.
Technological developments leading to re-thinking laws and regulations is nothing new, of course, and "we should outlaw employers hiring PIs to read every piece of text a human has ever posted online" is something that never would've been a realistic thing to pitch to a politician 25 years ago.
Now, the technology has changed, so we need to talk about it.
You seem to be agreeing that it would be bad, so like the original poster here proposed, let's focus on figuring out what "bad" use cases this tech makes far cheaper and more realistic today and then trying to convince others that we should regulate those use cases.
(The second thing I brought up is a difference not in scale but in deniability and ass-covering: it's illegal already to take a retaliatory action against an employee who complains about harassment, for instance. Figure out a sneaky-but-very-plausible way to make that employee just-happen-to-come-up-bad in some LLM-mediated part of a performance review process? Now you'd have a much harder time getting caught.)
Gotta fight fire with fire. For every public facing process we should perform bias analysis with specialised tools. These tools should elicit edge cases of all sorts, do automated "red teaming".
Putting on my Republican hat: it's too expensive. If we just focus on "was there bias in the output" the amount of time and effort required to analyze all the things is going to be super high. A completely-zero-trust economy would have far too many parasitical drains on productivity as we constantly have to prove everything to everyone every time.[0]
Taking off my Republican hat: which is why up-front regulation on specific actions and methods is what we need instead. Much easier for other people to spot, much more of a bright line "you did this thing, we said don't do this thing." Not at the granularity of "don't use ChatGPT specifically" but more like "these are the things we will allow and won't allow in how you process job application background checks" (we already do this for discrimination, I just think we should update it to reflect that we don't want a centralization/standardization of process so that people become de facto unemployable based on what some tool used by some company thinks about them).
[0] not to mention that looking for bias in outputs in a mechanical way is also gonna false-positive a bunch; p-hacking, but for accidentally getting sent to jail :(
LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.
I appreciate your concern, but the same issue would apply with existing expert systems or linear regression models that wouldn't typically be classified as "AI" today. Most power companies are subject to fairly strict regulations so they generally can't cut off customers unless bills are badly overdue, or the customer tampers with utility equipment. In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement; that may be objectionable on privacy grounds but that's a political issue, not really an AI issue. Where there's a large power imbalance between private citizens and large institutions we should address that with laws and regulations based on impact, fairness, and accountability rather than trying to specify how those institutions can use particular algorithms.
Unfortunately in the US if your law isn't that specific - and almost any law about "impact, fairness, and accountability" is going to be much harder to tighten up, interpretation-wise, than one about "don't fucking do these things" - then it's a big target for judges who don't like your definition of impact/fairness/accountability, or the definition of the agency you create to oversee it.
That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).
I don’t think it’s honest to suggest linear regression models fail in the same way current AI models do. Linear regression “fails” when you are an outlier case, whereas AI systems just occasionally truly fuck up in unpredictable ways on standard inputs.
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon.
This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic.
You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.
Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".
Well that's just the thing, the AI crowd spans a lot of opinions, from those who believe the endstate is a time travelling cyber Satan that will torture everybody and make clones to torture more people until the end of time, to those who fear large swaths of workers being replaced by an influx of heavily automated processes, to those who are worried about LLM-powered spambots destroying the notion of truth.
> those who fear large swaths of workers being replaced by an influx of heavily automated processes,
I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes.
My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI.
I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content.
Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work.
I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job.
I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy.
It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI.
Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.
Gee, it's almost as if everyone is trying to put the shiny new technology in their product because there's a hype/FOMO cycle going on. Remember when blockchain was gonna be in everything?
Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.
> those who believe the endstate is a time travelling cyber Satan that will torture everybody
This smacks of misinformation. If you're referring to that dreaded basilisk, very few people actually believe that scenario has any likelihood at all. Also, it has nothing to do with time travel anyway.
The other two things are more or less guaranteed because they're simple extensions of processes that are happening right now and can be seen all around us.
Yes, I listed it first because I think these guys are eating up all the oxygen in the room and the more plausible risks get kinda unfairly lumped with the crankier stuff.
(And apologies if I misled people into believing the superintelligence basilisk has time travel powers)
> Those "very few" people very prominently include Yudkowski himself, or so I understand.
I don't think that's true; he's stated before he doesn't and didn't believe it was true. His reaction was more about the stupidity of posting information hazards on public fora rather than that specific information hazard being credible. However, he could be lying post-facto.
And I do agree that it kind of sucks that Yud is the "face of AI risk" because it doesn't even matter how correct he might be; a lot of people just don't like him (I'm indifferent). There shouldn't be a "Face of AI risk" because there won't be a single person that everyone like. The idea is bigger than him.
Why does it need to be boring to be credible? Anything is non-credible if you arbitrarily label it to be "weird ... bullshit". That doesn't make your arguments sound.
Nanobots don’t exist in any meaningful let alone threatening way. So any ai scenario involving them is bullshit, and actually about nanotech not ai. That’s what you hear from the likes of Yudkowsky.
It doesn’t have to be boring, but boring sounds more likely to me, because it’s soemthing we don’t think much about… because it’s boring.
I mean it could though. The US military has drone swarms that can automatically search for and kill people with incredible ease. AI systems are already capable of killing tons of people through similar homemade devices. Just attach a gun and facial recognition to a drone and you've got a product that works.
Right, but there is a long way between that and civilisation collapse. Also neither of those systems have long planning or self improvement capabilities.
> Imagine we had started taking climate change seriously in 1990
I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then.
I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions:
The UN Framework Convention on Climate Change (UNFCCC), adopted at the 1992 Earth Summit is one of the first international treaties on the topic.
I remember one of the first conversations I had on "global warming" specifically and it was around 1996. But even before I had heard that specific term we were talking about losing large coastal regions due to rising sea levels. The rock band Tool released the album Ænema in 1996 and there is the constant refrain: "Learn to Swim". By the time Al Gore published "An Inconvenient Truth" in 2006, I was feeling the topic was a bit played out, to be honest.
At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do.
Define "very seriously". I know of cover-ups, mass propaganda, antagonistic lobbying, and political inaction. I don't really know of any very serious actions against global climate change predating the 90s.
> I don't really know of any very serious actions against global climate change predating the 90s.
There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier.
Although I was a child then, I even remember a lot of serious agitation happening in the 70s.
The Wright Brothers flew at Kitty Hawk in 1900. [0]
The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1]
We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900.
"Some people didn't understand the science, and so called it fiction" isn't a very strong argument.
But Bernoulli proved the scientific foundation of it well over a century before anyone actually did it.
No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.
What is so impressive about ChatGPT that it poses an existential threat? Many experts are highly critical (e.g. LeCun) and/or believe LLMs are nothing more than stochastic parrots.
Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent.
Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?
What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.
> Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?
It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility.
> Anyone who was working with transformers could have seen ChatGPT on the horizon
So, who did?
> [LeCun is] highly critical
I did a quick search and found this[1] trash-tier article explaining his views summed as:
> He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial.
I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth.
Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem.
Finally, your question:
> What is so impressive about ChatGPT that it poses an existential threat?
This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief:
> Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals.
So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow.
The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
> essentially 0%.
I sure hope you're right, but I don't think that's a common position to hold.
Hinton has been exaggerating the capabilities of AI for nearly a decade. As one example, if we listened to him in 2016/17 we would have an even worse radiologist shortage than we do right now.
> The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
It’s not assuming it won’t happen or burying our heads in the sands, it’s calling for measured responses.
My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
Autonomous AGI, whenever that happens, absolutely poses a threat. But it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
People assume advancements will continue to happen at their current pace. The last 6 years of advancements has been the product of scaling and refining self-attention + transformers from 2017/8.
> so who did
I’m not sure what you’re getting at. No one had the will/resources to burn billions on a language model that has ??? business plan, doesn’t mean no one saw it coming.
> Hinton has been exaggerating the capabilities of AI for nearly a decade. As one example, if we listened to him in 2016/17 we would have an even worse radiologist shortage than we do right now.
I can appreciate that. I haven't been listening to him for that long; having no idea what situation you're discussing, I'll take it at face value.
> My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
That's pretty reasonable. However, I will reiterate that this:
> it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
is not what I'm asserting. It's the unknowns which are dangerous. We're inventing something new in almost every way imaginable. It's not like we're talking about a new theoretical steelmaking process which could upset the economy, we're talking about the potential to build a new life form with godlike abilities. Alarm seems warranted to me. Sure, it might not happen soon... but also, it might happen soon. Shouldn't we be trying to prevent it, or control it, before it happens?
> People assume advancements will continue to happen at their current pace.
I think people assume advancements will accelerate, actually.
> Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.
In a way, all of those things have led to where we are now. And might I point out that a form of bioengineering may have caused the latest global pandemic? Not taking these things seriously because "they haven't killed us all yet" seems a little shortsighted.
> "not taking these things seriously" is very different from regulating something in it's infancy because of fear of the unknown.
I mean, we should have better regulated carbon emissions generations ago. We'd have a lot more time to deal with it, because we haven't been able to deal with it in the century+ we've known about it. And AI extinction risk is likely to move faster than carbon-related climate change.
> "not taking these things seriously" is very different from regulating something in it's infancy because of fear of the unknown.
I don't pretend to have the answer of what the balance is for oversight and regulation to optimize safety and innovation - but the first step for those responsible is to recognize the potential dangers and unknowns, and produce plans that can be discussed and debated.
> Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?
How about gain-of-function research? How about nuclear weapons? I lean libertarian but also recognize we have a responsibility to safeguard humanity.
> Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.
Did they? I've never heard about electricity creating fears about human extinction.
The industrial revolution does pose an existential risk: climate change.
Did bioengineering garner worries about existential risk? It certainly had and has a lot of people worried about various risks. And then lots of countries banned human cloning: https://en.wikipedia.org/wiki/Human_cloning#Current_law
> Evidence? This has been stated in sci-fi books from before I was born
Yes, many sci-fi authors of the past were very forward-thinking and predicted future technologies... that's sort of the point.
> and I don’t see any proof that we’re building something remotely close to this
Again, this is a fundamental misunderstanding of the risk.
1. We are building AI. AI is possible.
2. We are advancing AI. AI advancement is possible.
3. We have not built superhuman AGI. But superhuman AGI is probably possible.
The fact that AGI doesn't exist today is frequently argued as a reason we won't have it in the near future, but it's a non-sequitur. We will definitely have more advanced AI in 5 years than we do today, and we can't say what that AI will be capable of. Therefore, it's possible it will be AGI.
> The industrial revolution does pose an existential risk: climate change.
And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.
It’s true that once we understood the concrete mechanisms and could evaluate alternatives, not mitigating the risk has been irresponsible, but that was…significantly later than the Industrial Revolution itself.
> And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.
Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change or any other existential threat. It might have made us more vulnerable to natural pandemics... but it also would have slowed the emergence of natural pandemics and virtually prohibited artificial pandemics.
> Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change
Neither do I, but then, 100 years ago doesn't clearly meet the description, anyhow. The discovery of the greenhouse effect and CO2’s role in it was after not very long after the dates usually used for the end of the (first) industrial revolution (which is usually what is meant without modifiers, not the second—fourth industrial revolutions) and well over 100 years ago.
> Replace "100 years" with whatever timeframe is perfect for you and I'll make the same argument
At any time before we had the information to understand thr mechanism of problems and evaluate mitigations, its very unlikely that we would choose tbe right mitigations. Sure, if we choose thr mitigations that we would choose based on information today before we had that information, that would be great, but its at least as likely that any attempt to mitigate risk of unknown mechanism vefore we had that infornation would have done exactly the wrong thing.
Basically, “Well, I have information today by which I can design a policy which would have been beneficial if implemented before that information was available” is not an argument that people seeking to mitigate a vaguely imagined risk without any understanding of its mechanism would have the means to design a productive intervention.
I'm saying that with what we know now, it's clear we could have made different choices that would have worked out better for us earlier in industry.
You're saying that the information at the time made it impossible to correctly make the optimal choice. (By that I mean, making the optimal choice would have appeared to be the wrong choice, based on information available.)
I don't agree with your point; I think we knew that burning carbon was going to change the atmosphere for the worse and we could have been more careful by simply burning less carbon. However, I concede that it's unknowable what science and technology would look like if we had done that.
In analogy to AI, that is very much a point of conversation in alignment/safety discussions. Should we wait until we have better tools to figure out alignment, since it's so insanely difficult to figure out now? Frankly, nobody knows for sure the answer to that question either. My contention is that you (and others) are assuming the answer to that question is yes, but that's not necessarily correct, even based on all the information and analogies we have now.
It hasn't been proven possible, but it's reasonable that it's possible. There's no theory I know of saying that it's unlikely to be possible. Compared to FTL travel, for example, which we have theory enough to state is not likely to be possible.
You can conjure fantastic (positive or negative) unknowns about anything. Unless, for the negative case, you have a concrete reason for believing some class of net-adverse events are probable along some particular route and some concrete basis for evaluating whether alternative actions will both mitigate that risk and not create greater new ones, you have no rational basis for acting based on those unknowns.
> You can conjure fantastic (positive or negative) unknowns about anything.
It's not something conjured, though. Unless you believe superhuman AGI which can set its own goals is impossible, or that it's possible but it can't possibly have goals that pose a threat to humanity, then it's reasonable to believe we can invent superhuman AGI which threatens our own existence. Technologically, I think it's possible to do so.
So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible? I think we are. Even though we're mostly focused on LLMs right now, and even if LLMs are not a path to AGI (which I agree they probably aren't on their own), it's not like there's no research happening on other pathways. Additionally, we can't be sure that current LLM research won't contribute to AGI - maybe the real path is a synthesis of multiple forms of AI.
So there are some very specific unknowns standing between us and possible doom. We're not making up fantasies wholesale.
- Dangerous AI is possible
- We're racing to create it as soon as we can
- We don't know how to tell if we've crossed a threshold into having dangerous AI
That last point is an known unknown which is extraordinarily dangerous. It's like if you're blindfolded and moving your hand toward a running chainsaw. You know that if you move your hand far enough it will be shredded, you know you're moving your hand forward, you just don't how close your hand is to the chainsaw. You should be concerned in that situation.
It is something believed without any evidentiary basis which enables rationally weighing the actual risk or evaluating mitigation strategies so as to weigh their probable benefits against their probable costs and the associated changed in the overall risk profile; I'm not suggesting literal magical incantations when I say “conjured”.
> Unless you believe superhuman AGI which can set its own goals is impossible,
Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not. But I don’t actually know that this is important, or even that the mucj weaker assumption that it’s goals might not exactly represent human intent (which is definitely possible of AGI itself is) is important, since human intention can itself be an existential risk.
> or that it's possible but it can't possibly have goals that pose a threat to humanity
Its definitely possible for an intelligence, even a human one, to have goals that are a threat, including an existential one, to humanity. That’s an existing x-risk without superhuman artificial intelligence. It may even be mitigated by superhuman artificial intelligence (as might other x-risks like “earth gets smashed by an asteroid”), if it is possible for such to exist without goals that are antithetical to human survival, in which case delaying superhuman AI could increase existential risk.
But, while this is all nice and conceivable, we have no way of balancing the net impact, positive or negative, of any course on AI to existential risk due to friendly or hostile superintelligences.
> So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible?
No, I can't agree with that, as it would require that every resource that could advance superhuman AI, and specifically dangerous* superhuman AI, was being directed as effectively as possible to that end. I don’t think we are anywhere close to that. Because AI has some mindshare beyond immediate economic utility, we’re probably heading toward AGI, if it is possible and if our current understanding of AI isn’t so wrong as to make our work irrrlevant even in eliminating bad routes, slightly faster than immediately-apparent economic incentives would promote, but nowhere close to as rapidly as possible.
> It is something believed without any evidentiary basis which enables rationally weighing
But that's exactly true of either position you take. Imagine I'm arguing that aliens probably exist and you're arguing they probably don't. There is no such thing as a belief based on evidence in this case; we only have logical reasoning and assumptions and beliefs.
> Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not.
This feels like moving the goal posts... all it needs to be able to do is make decisions that are dangerous to humanity. I felt that definition was obvious. All it needs to do do is mimic an oil executive, because, as you say, human intent is also an existential risk. Imagine an oil executive that could get the upper hand in all possible negotiations and desired to make the entire global economy subservient to the needs of oil production.
> delaying superhuman AI could increase existential risk.
Oh, yeah I totally agree that we can't truly know one way or the other. But it kind of feels like deciding to invent and deploy nuclear bombs as a deterrent to war. Did it end war? Absolutely not. Many would argue it didn't even that specific war. We are of course fortunate that nuclear war has not yet destroyed civilization, but a) there's still time, b) we aren't necessarily better off than if we had no nukes, and c) not all risks are equal - AI could be more dangerous.
> as it would require that every resource that could advance superhuman AI
Actually, you're right. I was definitely overstating our progress and didn't think through that argument clearly. But maybe it would be safer if we were devoting all global resources toward a coordinated AGI alignment and development effort...
Because I see nothing that indicates it is. It's pretty hard to come up with evidence for absence, though, which is why the burden of proof is on those making the positive claim, not on those who are hearing the positive claim.
All we're talking about is logical reasoning. There is no burden of proof in the way that you're using it (and other keep using it). I'm not claiming that mythological gods exist.
Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.
I expect that it's possible for superhuman intelligence to exist. I don't think I need to prove that it's physically possible for a superhuman intelligence, natural or otherwise, to exist. Do you think it's not possible? If so, I'd say you need to prove why you think human-level intelligence is literally the pinnacle of intelligence that is physically possible in the universe (especially when narrow AI are already superhuman at many tasks).
Artificial intelligence exists and can be created by humans. This could be disputed if you wanted to gerrymander some really specific borders around what "intelligence" is, but if we don't agree that artificial intelligence exists today, just say than and we'll be done talking.
AGI does not exist. I assert that it can exist, because narrow AI can be improved. Do you believe it's not physically possible for human-level AGI to ever exist? If so, you need to prove to me why we can create narrow AI that is superhuman but cannot ever synthesize them into a human-like intelligence. (And also please give me the scale by which you're measuring an alien brain against ours.)
And if we both agree on all the above, then I take my assertion further: It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI (in fact, I think that if it's possible to create AGI which matches humans exactly, it's probably easier to surpass human levels than to match them exactly, because it's hard enough to measure human intelligence).
So if it's physically possible for us to create superhuman AGI, what exactly am I proving? Do I need to invent the technology in order to prove that the concept is possible?
> Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.
Of course.
> I expect that it's possible for superhuman intelligence to exist.
I agree.
> Artificial intelligence exists and can be created by humans.
This is where we disagree, although as you say, it depends on what you mean by "intelligence". I think in the context of this discussion, we're talking about some sort of superhuman intelligence with its own will and perhaps consciousness.
In the largest possible sense, it's probable that creating such a thing can be done. Where we probably disagree is that I think we're a very, very long way from being able to do that.
Nothing we have now makes me suspect that we're anywhere near that sort of thing.
> It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI
Agreed. But I assert that we're nowhere near being able to create human-level AGI right now.
> in fact, I think that if it's possible to create AGI which matches humans exactly
I think this isn't possible at all unless you also create an entire human body.
> what exactly am I proving?
The assertion that AGI is a thing that is likely to happen soon. Personally, I think it's vanishingly unlikely to happen within our lifetimes, and very unlikely within the lifetime of our children.
Freaking out about the risks of AGI right now is a serious overreaction. Calmly pondering it would be more appropriate.
All that is pretty reasonable, I appreciate your measured response.
So the fundamental disagreement is over how close we are to dangerous AI.
If you're right, we might delay useful AI by decades or centuries being overly cautious. Useful AI could either alleviate untold suffering, or it could cause untold suffering and inequality by misuse.
If I'm right, we could go extinct for lack of caution.
And given that we really don't have any way of knowing with any amount of certainty how close we are to AGI, you see why people are concerned, right? Calmly pondering feels like a serious underreaction. Vigorous, frequent, and open debate would be more appropriate. I'm not saying let's turn off all our computers for a decade.
> When we improve LLMs, won't they just be a fancier auto-complete?
Not necessarily, and if so, only if your first postulate is true. Which is really difficult to say because "fancy auto-complete" is too vague to argue. Human speech can also be described as fancy auto-complete.
Luckily we don't have to argue that because I specifically am saying it doesn't have to be LLMs. You've missed the point entirely.
When it comes to the question of when strong AI will come, I think our society's reaction to ChatGPT is more worrying than the technology itself. Namely, because of ChatGPT, we see major companies, governments, etc clamoring over AI today like never before. The magnitude of this hype and its resulting funding rush is unprecedented. Nobody could have predicted that this would have happened so quickly.
>AI had a 1-in-10 chance of shutting down all power generation on earth.
Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.
You mean an industry with so much inertia and hesitancy they still run mission critical systems on tech from the early 2000s?
You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
I do, yes. The thing is that the power company as a whole is not a decision maker. Rather, individual people with short-term horizons are. If you can save the company some money, make a bunch of money yourself that way, and then get out, you'll have done well for yourself and you can ignore the consequences, or you can convince yourself that you'll be able to ignore the consequences, and that will be enough for plenty of people to want to do it.
>You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
Their technology conservatism is either because they want to use old reliable tech, or because they're unable to do hard things. If it's the latter then we have a problem.
GPT4 has been a great teacher, I love it, but I think a lot of people and industries are already "faking it" and they'll be happy to turn any decision over to AI as long as it makes them look good in the short term. We won't see this from the outside either, like the lawyer who let GPT secretly do his work, others will do the same, but won't get caught like the lawyer did.
Due in large part to their laziness with equipment... which is to say, they'd probably take up AI if they thought it could save them on the salaries of the people who might tell them they equipment should have been replaced 30 years ago.
Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth?
Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs.
I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
> Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth?
No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to:
> I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning:
1. "LLMs are an amazing technology [which advanced at an unexpected rate]"
2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research
3. Such acceleration could conceivably happen or be happening now in other forms of AI
4. The path to AGI is COMPLETELY unknown.
5. The origins and structures of consciousness and agency in intelligent systems are completely unknown
6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life.
7. AGI is inherently alien and there's no reason to expect it to think the way we do.
8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc)
So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
Yes, but we also don't have reason to believe summoning alien entities through any kind of rift is at all possible. According to known physics, it can't happen.
On the other hand, we don't have any scientific reason to believe that superhuman AGI isn't possible. We know AI is possible, because we have it right now. What's preventing it from getting smarter than us? The human soul?
To clarify, most believe we likely have started on the path to superhuman AGI. We just don't know how or when we'll reach an arbitrary threshold where it becomes dangerous. There are many finish lines and we can't see any of them, but we're running as fast as we can down the path of least resistance and hoping we don't cross a finish line we don't like.
> we don't have any scientific reason to believe that superhuman AGI isn't possible
Sorry, what? According to which scientific theories is AGI possible, and what principles are those theories based on? What testable hypotheses say that it is, and which experimental evidence has been produced in favour of AGI being possible?
We have no evidence one way or the other, but it's kind of insane to me to think that superhuman AGI is not possible.
Human-level intelligence is possible. See: humans
Do you think it's physically impossible to recreate human-level general intelligence artificially?
Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Is there any physical law whatsoever that even hints that intelligence should be able to create AGI at its own level, but in no way more intelligent?
I never said there was a physical theory that predicts superhuman AGI. But in contrast, there are other concepts such as time travel which are prohibited by certain theories we have. It seems very unlikely for time travel to be possible. Superhuman AGI is not prohibited from existence.
> Do you think it's physically impossible to recreate human-level general intelligence artificially?
I have no idea. I have no basis for concluding it's either possible or impossible.
> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
No, I think a godlike being could materialise an entity with superhuman intelligence, or natural selection could evolve one (perhaps even on earth -- perhaps it already has!).
> Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Even assuming I believe both the above are possible that doesn't mean I think it's possible they could occur on planet earth within any meaningful time frame. I also think that interstellar travel is compatible with the laws of physics but just because we've sent humans to the moon and robots to mars we have no idea whether interstellar travel is feasible or when it might occur. For me the response to the advent of GPT is like people saying "wow, we've sent men to walk on the moon -- if we keep going people will soon be visiting mars/alpha centauri!".
There's no rational basis (scientific, logical, anything) for knowing whether AGI is more akin to mars in this analogy (and we're really quite close) or to alpha centauri (and we're nowhere near).
>> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
> No, I think a godlike being...
Superhuman intelligence is a reality in many fields already: chess, Go, self-driving cars at certain conditions, calculations and many more. This is not necessary to have it for everything at all.
I'm not really sure what you're saying, but sure, we've had superhuman intelligences for at least 200 years, when Babbage invented the difference engine. If you're willing to count abaci we've had superhuman intelligences for thousands of years. You might object that the intelligence of an abacus is manifest not just in a device but in an amalgam of human and device. But isn't the same true for all computation that humanity has so far harnessed?
I'll take researchers' gut feelings more seriously once we're out of the ongoing hype cycle. Right now most people are probably overestimating applicability of current advances, and that colours their predictions. Researchers are just as affected by this.
And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.
> unless you believe that AI can't possibly pose an existential risk within the next couple decades or so.
I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one.
> Actually I'd love to know what your estimate actually is for AI becoming an existential threat
My estimate is: never. At least not in the form of some superintelligent AGI.
A power company is unlikely to do this, in part because they are not an industry that fetishizes growth at any cost with the value of individual users approaching zero. And in part because in many markets, they're regulated and need to provide service.
But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.
I agree that’s a risk, but it doesn’t seem like a different magnitude of risk than power companies using other crappy software to control their systems. It might break a few times, which will be annoying, but it won’t lead to the immediate end of civilization. They can use the risk handling mechanisms they’ve always used for infrastructure, which seem to give acceptable results.
According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.
The way I see it is that the existential risk comes from humans losing control. Here is a very abstract argument for why AI is relevant here.
Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen.
AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control.
The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different.
The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.
> AI is likely to cause total extinction the first time something goes wrong
Maybe that is an argument you've seen, but the more convincing argument I know of is that we have no way of knowing we've gone too far until after the fact. Doesn't matter how many times it goes right first, we don't even know how to determine if and when it has gone wrong. A powerful enough AI could deceive us into thinking things are going "right".
The Yuddites are going for purposeful exaggeration because discussing things in a nuanced manner won't get support quickly enough from people in power. Fear is a stronger immediate motivator than curiosity or distrust. The reality is that multiple AI systems will likely contradict eachother leading to systems collapse, killing thousands of people as power, transit, or trade systems shut down. It is of course a massive concern, but it isn't as scary as total extinction, and the fear is destruction on that scale will be ignored as an anomaly or simply an operation risk.
You have an extremely high risk dying in a vehicle collision but people dismiss the risk because it isn't immediately apparent that 40,000 people die every year in the U.S. from that very cause. Certain deadly events become common enough that they get ignored due to desensitization, or become categorized as individual and anecdotal tragedies because they aren't mass casualty events like a plane crash or a hotel collapsing. If a few thousand people die every quarter from AI lockups or failures that nobody could've predicted or prevented because the AI is fully operationally autonomous, it will be treated the same as vehicle deaths. And that is the true fear.
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
Many of those people were not tricked, tech CEOs are involving themselves with this and picking up Yudkowsky's particular strain of AI safety intentionally.
Instead of focusing on the very real problems AI causes and exacerbates today, problems that might affect tech companies' bottom lines if addressed, CEOs can control the narrative and appear proactive by buying into this brand of science fiction.
Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.
> Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.
The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews.
I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent.
I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.
We play with tropes based on appearances within our culture.
Consider Shrek. The story is made to subvert the trope that the ugly person is the villain, as it was in Disney movies.
But the real point isn’t some Platonic view of Beauty. It is about expectations. Like how advertisers use people wearing lab coats to subliminally feed credibility into the words of actors.
You may wish to be free of such bias, and you may even work to raise your own awareness and strive to critically examine the message of people. And let us all work towards such a world. But we should not pretend such a world exists today, despite our desires. It is a world that may exist in the future once we mature as a species.
Until that day arrives, advertisers and political movements will use our biases against us.
My claim isn’t that yudkowski is ugly, it is that he represents a trope that is being used to bias his message.
He didn't decide a whole lot about how he looks, except he wears a hat. His hair color is just how he was made, same with how his face looks.
He appears to make an effort to look good just like most everyone else. If he was trying to look a certain way he might have tattoos or have died his hair. He doesn't, that I know of.
You seem to be policing a small recent decision he made to wear a hat.
I didn’t create the meme of a neckbeard wearing a fedora. Know you meme entry [1] is 9 years old at this point.
Just like I didn’t create the trope of a scientist wearing a white lab coat. But if someone is doing a 2 min video sketch of a scientist, it is a good visual aid to include a white lab coat to communicate to an audience the role of a character.
And in the same way, a top hat would communicate a certain kind of character in a movie.
In the same way, there is a communication of a particular kind of character with other hat choices.
Allowed? Where do I say he is or isn’t allowed? You must be trolling to so poorly represent what I’m saying.
He can wear anything he wants. So can you. So can anyone.
If he wants to, and for all I care, he can dress up like a clown. Or dress up like a jester. Or he could wear Gucci or Prada. Or wear steampunk leather outfits like he is in the world of mad max. Or he could go clean cut preppy. He can wear a fedora and a trilby on top of one another.
But, by personal choice and temperament, he wants to dress as he dresses. It just so happens that his personal preference aligns with a particular trope. A particularly uncool trope in popular culture.
I’m pointing out that alignment. I am not making him dress that way and I am not forcing culture to judge him.
I'm not buying this alternative reality where I'm trolling by pushing back on mocking someone's choice of a hat that is one of a limited number of common hat styles.
It's reality, and as much about mannerism as looks. OP wasn't attacking chomsky on a personal level, just pointing out how he's often perceived.
It's sort of unfixable because the taste-makers at NYT and WaPo sorted themselves into the conformist camp so early in their lives that anyone outside that camp is trivially portrayable as "weird" to them.
It's interesting how the people who pride themselves on how they're speaking up for the weak are the first and most vicious in bullying acceptable targets.
Yud on one hand believes that AI is an actual *VERY LITERAL* existential humanity of which it is completely 100% paramount to get the rest of humanity on board with preventing our own destruction.
And yet he doesn't think it is worth the effort to take off his trilby hat in order to come across as marginally more respectable to the average person? When the way he is choosing to present himself is undermining his own message then yes, it is worth bringing up.
OP implied that Eliezer and Noam are one of the ugliest people alive. I'm very surprised so many people defend that comment and pretend it is relevant and discussion worthy.
Yud is a doomsday cult leader (https://archive.is/YTqlI) who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse. Although they call themselves "rationalists", I prefer to refer to these extremists as ℚ-anon. Their otherwise dismissible ravings are being converted to whitewashed publication friendly versions and fanned by commercial actors in an apparent effort to secure a state backed monopoly on technology that is otherwise relatively moatless, potentially to the peril of AI researchers (including ones that work for them) to the extent that anyone takes them seriously rather than using them as a plausible excuse to hand out windfalls.
We don't have an AI safety problem, we have an human safety problem. Violent extremists are unsafe, greedy corporations seeking to suppress the free exploration and exercise of mathematics are unsafe. Existent language model chatbots are primarily just a public relations risk, at least so long as the government doesn't step in and create a monopoly that gives the market leaders choice of training biases the force of law.
This whole subthread is both silly and sad. So suddenly Yudkowsky is the Thought Leader and a punching bag when people need to dismiss the entire AI risk perspective. A week or two ago, during previous round of statement letters and signatures, Yudkowsky was apparently a Complete Nobody. Anyway.
Are you really using that link to support your argument in the earnest? You know this was posted on April Fool's day? The text itself mentions it near the end, and it's also explicitly tagged as such directly below the headline/byline. At least warn people who're going to skim this.
> who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse.
You mean that bit from his article in Time Magazine? Where he talked about airstrikes against GPU farms might be necessary in case some country/group is violating the international moratorium he's proposing? I don't recall him advocating mass murder there, or ever. Of course, since he is insisting on a strong international agreement, military intervention - yes, including nuclear weapons - is part of the picture, because that's literally how international agreements are ultimately enforced.
As for the rest - the guy has been saying the same thing for way over a decade now. The mainstream interest only really picked up in the last half a year. But somehow he's in on it.
> the guy has been saying the same thing for way over a decade now
indeed, and by many who saw it regarded as a kookery, not justifying of much concern. If he wanted to preach to his S&M dunegon about the big bad machines that will punish you for all eternity, well not my circus, not my monkeys.
Thus far his followers who've turned violent have only directed it at their extended group [1] for not doing enough to stop the end of the world (and that one murder and attempted murder of their landlord [2], but I'm willing to believe that was a chance event and not a trend).
I now consider it an actual concern now that the profile has been expanded so dramatically there is a much greater risk that someone will fall for ℚ-anon who isn't so distracted by their S&M dungeon that they actually find time in their busy schedule to 'save the world'-- just like Qanon was some internet joke until the wrong person fell for it and rolled up to
Comet Ping Pong and opened fire. That person only thought they were saving children from being sex slaves, whats someone powered by the belief that they are saving the lives all current and future children going to be willing to do? Their prophet says even nukes would be justified.
[And not just their lives-- there are fates worse than death, some of their scripture involves fearmongering about malevolent AI's simulating trillions of copies of people and subjecting them to the worst torture imaginable, just because they didn't do everything in their power to bring the malevolent AI into existence. Lets not even get into the fact that their proposed solution--since malevolent AI is inevitable unless we abandon advanced technologies "just don't do it" is only a short term fix at best-- to create an AI god embodying the values of the faithful so that it can permanently suppress all competing machine intelligence and engineering human life to 'optimize' away suffering.]
Maybe you live far enough away or don't have anyone you love peripherally connected to AI enough that you're not concerned that they'll be murdered due to this insanity, but I am a little worried and right now the best way I can think to mitigate the risk is to bring people's attention to the fact that they're normalizing and legitimizing a mentally ill position that expressly justifies the use of violence (and not just in "ha ha only serious" April fools posts) against a fanciful apocalyptic threat from the fevered imagination of a cult.
> Are you really using that link to support your argument in the earnest?
Yes, the post was presented as a joke but it legitimately reflects his position, as also reiterated in interviews and articles.
Moreover, it's a "rational" response to those who actually believe the claimed threat, particularly from the utilitarian ethics perspective advocated in ℚ-anon circles: they hold that it's ethical to murder a million to save billions (or trillions 'discounted to present value', counting all the humans that will never exist due to extinction-- "What matters most about our actions is their very long term effects.").
> I don't recall him advocating mass murder there
Nuclear weapons have been used twice in combat in human history. In one instance the death toll was a hundred and twenty nine thousand people, in the other it was two hundred and twenty six thousand people. The nuclear weapons in the US arsenal today are many times more powerful than those early weapons.
It's not like data centers are not blast hardened targets, no one has described NAFTA, the Paris agreement, or the berne convention as being enforced by nuclear weapons. In a serious publication with a readership of millions Yud unambiguously advocated using the threat of mass murder as a deterrent against parties performing too many mathematical calculations.
How many of his new followers will agree with the widespread death of a nuclear weapon a...
> So suddenly Yudkowsky is the Thought Leader and a punching bag
I only meant to imply the latter. I suppose I inadvertently applied the former by mentioning him juxtaposed to Noam Chomsky and Michael Moore, although it is a stretch for me to even see those two as "Thought Leaders" in the present day. Chomsky is some weird kind of anarcho-syndicalist and Michael Moore seemed to fade away during the Obama years. Neither are really "leading" anything these days.
> A week or two ago ... Yudkowsky was apparently a Complete Nobody.
There is nothing contradicting being a punching bag and a complete nobody. It's like a D-list actor whose entire career is to get his ass kicked by the hero in action movies. Or the B-tier heel in professional wrestling. You might eventually recognize his face and he may even achieve some cult-status fame. Their whole purpose is to display the might of the eventual hero so we often allow them to embody some apparent strength so that a sense of risk for the hero is maintained.
Yudkowski makes for a juicy target. For example /r/sneerclub has been around since 2015! The fact that the mainstream media has now identified him as a magnet for derision isn't surprising. No one has to know who he is. They just take one look at him, hear a couple of sentences from his mouth and they immediately form a strong opinion on his character.
> The mainstream interest only really picked up in the last half a year. But somehow he's in on it.
What do you mean "he's in on it"? I'm suggesting he is an unwitting participant. I have no doubt he actually believes he is the main character. He doesn't lack the ego for that.
FWIW, even though I personally really dislike Yudkowski as a person - I believe he has some interesting points to make. And frankly, it bugs me that he is so often dismissed without deep engagement with his arguments. But he is so lacking in media training and social skills that he is easily misdirected to look like the exact foolish/cartoonish caricature the media wants. Even in friendly podcast situations he comes off as unhinged.
And I honestly believe that boosting the signal of characters like Yudkowski is part of the mainstream establishments playbook for dealing with dissent. In comparison, Sam Altman's "aw shucks" performance in front of the senate, with favorable sound bites where he boyishly claims to have no financial stake in OpenAI, was positively wholesome.
> Yud is a doomsday cult leader (https://archive.is/YTqlI) who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse. Although they call themselves "rationalists"
No, it's also the AI. There are two broad categories of horrible outcomes:
1. Bad people control and use AI to dominate/destroy the world
2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just
1. We might create ML models smarter than ourselves soon
2. We don't really understand how ML models work or what they might do
3. That seems dangerous
There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.).
The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.
Counter arguments to what exactly? Your line of thought is: some advanced technology is potentially dangerous. This is so vague, how can anyone counter argue? The sun is dangerous, water can be very dangerous, even food! I’m not sure I can follow.
It is on the AI-doomers to explain why sci-fi concepts like “AI improving itself into a super intelligence” or “AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead” have any relevance in the real world.
The entire AI-doomer worldview is built off of unproven assumptions about the nature of intelligence and what computers are capable of, largely because thought leaders in the movement are incapable of separating sci-fi from reality.
Paperclip maximizer is the wrong word. Let's call it the "engagement maximizer" and you have a pretty accurate statement about how our civilization will be ended by use of algorithms and eventually AI to blow up our shared sense of culture.
If you watched the conversation around Meta with Facebook and Instagram, and then later the conversation around TikTok, almost nobody cut to the heart of the issue, which is that "algorithms" are being used to make decisions about what to show people, and that the algorithms themselves have been changed subtly hundreds or thousands of times over the years to maximize engagement, until the type of engagement being maximized has made people basically crazy. The same engagement-maximizing work will proceed with AI, and it will allow the software engineers and managers responsible for developing the "algorithm" abdicate even more responsibility for genocides and for mass hysteria because they can pretend like they have no control over the algorithm.
The same irresponsible people will pocket a bunch of money for all this work and they will maximize engagement until it blows up our entire culture. And because nobody understands it except the techno-cultists, nobody will hold them accountable.
"Engagement maximizer" also aligns with descriptions of the anti-religion of the future I've seen. What does engage people the most? Romans knew it's bread and circuses, where circuses are bloody cruel shows on public stadiums. Today the most engaging activity is onlyfans and the like: the other dark side of human nature. Dial both to 11, add an all seeing AI with merciless thought police and you'll get an accurate picture of 2450 AD. The only variable is how long that grim stage of society will last before the tech breaking down.
I don't 100% disagree with you, but I'm still uncomfortable with this line of thinking. It smacks of "the plebs don't know what's bad for them, but I do".
I find it hard to fault someone who just gives the crowd what it wants.
I don't think it's so much "the plebs don't know what's bad for them, but I do" as it is "These people know something's wrong but haven't been given the vocabulary, tools, and education necessary to spot the danger as easily." Many ordinary people have lapsed into willful ignorance or apathy because the burden of learning the dangers is too high for them with so many other common problems going around these days.
Yeah for me it's more like this. I'm totally fine if people make the conscious decision to trade engagement for stimulation. But what I'm very concerned about is just how many people are forced into this situation because the network has captured their friends. One of the most concerning things for me is that in studies of how damaging social networks are for teenage girls, girls who were off the social network were markedly better off except if all of their friends were on the social network, in which case they were worse off. Companies like Meta are exploiting these network effects, which are harmful, to make people dependent on their platform in a very "addiction-like" way.
Hell, so much so, that you can get awards, probably even tenure, for writing a compelling argument or providing technical research for how we could guarantee that we'd be able to turn off a super intelligent AGI. Thus far: no solution.
We have however found that even smart people initially think "just turn it off" will work.
Neil deGrasse Tyson used to think you could just turn it off. He has since changed his mind. So does every public intellectual I can think of that engages with the arguments in good faith. Even the OP article concedes AGI research into the possibility that AGI could kill us all is important (just that other things are more important right now). There is no argument made that "we could just turn it off" in OP and therefore we shouldn't do the research at all.
> AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead
No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.
> The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives.
This isn't an argument, it is an assumption.
AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true.
> As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.
The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.
It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.
The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception.
The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.
Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent.
> The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.
It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy:
Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable.
Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us.
> It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.
I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.
> I have seen people make what is effectively this argument.
Ok, well I think they're foolish.
> That's a sci-fi trope. There is no reason that it must be the case.
No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so.
> That would be more relevant if the ants had created us to be the way we are.
sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.
The paperclip maximizer is an extreme thought experiment to make a point about the the orthogonality thesis. If you think anyone actually takes it seriously as a real-world thing you never understood the example. It is used to guide discussion.
The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing.
If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: https://www.youtube.com/watch?v=ZeecOKBus3Q
Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments.
Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.
None of those arguments for existential risk are actually "clear" or "technical". Just a lot of hand waving which only impresses those who don't understand the technology.
You're not even asking the right question. No one can prove a negative. Extraordinary claims require extraordinary evidence. So far no one has produced real evidence that the latest AI developments represent any sort of existential threat. The proponents are essentially making religious arguments, not scientific ones.
It's also demonstrable that we have other physical ideas such as FTL travel that are indicated to be impossible by current theories of physics. If we didn't have the math saying otherwise, it would be an open question of whether we can travel faster than light; but we have pretty solid math math saying we cannot.
And what we're talking about is logical in nature. Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes. Therefore, how is it reasonable to say that creating human-level artificial intelligence should be assumed impossible until proven? Just because we don't have the technology for it yet doesn't mean we should assume it's impossible. Logically, it follows from the first two assertions that it is physically possible to create artificial human-like intelligence.
Once again, and I don't know if I've said it in this thread directly but I've had to post it over and over again, nobody is claiming that the latest AI developments represent an existential threat. That's a complete mischaracterization of the debate and reeks of bad-faith argument but I will give you the benefit of the doubt and assume you've misunderstood.
In fact, that statement conflates two things into one nonsense argument. What's scary about current AI is how quickly it is moving, indicating a short timeline to (future) dangerous AGI. Also, it is possible that future AGI will be dangerous. These are two separate, rather simple assertions. If you disagree with either, that's fair, but you have to address them separately.
No, that's not right at all. You're just making things up and raising points that are irrelevant to the issue at hand. There is no logic in your claims. In particular there is zero evidence that current AI indicates a short timeline to future AGI. What a load of crap.
In logic, sure. But we're not living in a system of formal logic. We're living in a very messy world, full of physics, chemistry, and even (shudder) biology.
Here's the important question:
What would you consider to be sufficient proof that AGI is impossible?
Like, hypothetically. Doesn't even have to be based on any of the current facts on the ground in our universe. What facts or arguments could possibly convince you that this is not something that can ever happen?
If the answer is "nothing that I can think of", then you're asking other people to provide something you can't even define.
(If the answer is "nothing, definitely", then that means AGI is, for you, unfalsifiable, and essentially falls into the same category as religion.)
> Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes.
And here, you're falling victim to the ambiguity in human language (or, at least, in English).
"Intelligence" is not a clearly-defined word in this context, and while you seem to be presenting it as meaning the same thing in those two sentences, I would claim that it does not.
In the second sentence, it is clear that it is intended to mean "thinking intelligently, in a manner and to a degree similarly to humans".
In the first sentence, it cannot mean anything about "thinking in the same manner as humans," because you are talking about "artificial intelligences" that have already been created, and none of them think in anything like the same manner as humans. The difference between existing "artificial intelligences" and either humans or a hypothetical AGI is a difference in kind, not in degree, and you (and many others) gloss over that when you talk about "artificial intelligence" in one breath and "human-level intelligence" in the next.
Saying that all we need to do is keep going on the same track we're on with LLMs and similar "AI" programs, and we'll very soon (or ever!) reach AGI, is very like saying all we need to do to solve NP-hard problems in P-time is to throw more hardware at it. Sure, you'll get faster at doing the thing you're doing, but without some hitherto-unforeseen breakthrough (proving P=NP in the latter case; figuring out how to make AGI in the former), you'll never bridge a difference of kind by increasing the degree of effort.
> what would you consider to be sufficient proof that AGI is impossible?
This is trivially easy? People used to believe FTL was possible. Then our understanding of physics changed and we now understand it is an impossible limit to pass.
Would you say people who believed FTL was possible before physics research showed it to be impossible were believing something "religious" and "unfalsifiable" ? Please, they believed something totally within respectable epistemic parameters given what they knew at the time. In fact, the speed of light as an upper limit to speed seems very counterintuitive and "silly" at first glance. Why should a limit exist?
Sure, it might well be some fact about intelligence that we wont be able to get to AGI ever just by throwing more compute and layers at it for the next few decades or any time soon given current technology.
But nobody has come up with a slam-dunk, with empirical backing, that indeed there is some limit and we won't ever get AGI despite current trends. In fact, the opposite has happened and people like Geoffrey Hinton who used to believe AI risk is fanciful and AGI is a long time away has changed his mind, given current trends. We don't have research giving us that FTL limit, so why believe the limit exists? Why do you believe the limit exists? Or do you believe the probability is so low that we shouldn't worry? Ok, what probability do you place on AGI being created in the next 100 years and what would you have to see such that your probability crosses some threshold that it makes sense to worry about it?
(PS: If aliens suddenly arrive and they have completely alien psychologies such that when we discuss our relative intelligences it makes sense to talk about it in kind and not degree, I really don't think anyone is going to care about this distinction. What's important is how well can these aliens achieve their goals relative to us. And if they can achieve any goal of theirs that comes in conflict with our goals then we can reasonably say that are more intelligent than us)
Geoffrey Hinton is a fool. Despite his academic credentials he is deeply ignorant of basic technology and his predictions are not to be taken seriously.
I didn’t make the technical arguments, that would be too long for an HN comment. Check out Robert Miles’s YouTube channel for a good introduction to the more technical side.
You tell them that this is the greatest danger to humanity of all time and that they are uniquely suited to averting that danger and they don't have to change or sacrifice or risk anything in their life while fighting this danger.
It's a very compelling combination of ego and convenience.
> If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk.
To me it doesn't feel technical at all--just superficial use of some domain verbiage with lots of degrees of freedom to duct tape it all together into a story. He very much reminds me of Eric Drexler and the nanotech doomerism of the 80s and 90s. Guy also had all the right verbiage and a small following of fairly educated people. But where is the grey goo?
If we need a counterargument to Yudkowsky do we also need one to Drexler?
While I don't agree with Yudkowsky's over hyping of the tech as well. This stance of slowing down its proliferation in infrastructure matters is also very limiting.
In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate.
Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods).
The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.
This is going to sound flippant, but I'm serious: in a world of nonsense, something that generates nonsense (ai) is a fantastic tool.
The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong.
Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai.
I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?
This is 100% the right view. The biggest AI danger is relying on unreliable stochastic systems for automated decisionmaking, resulting in some kind of kafka-esque nightmare rather than something flowery like human extinctions.
This is akin to saying the risk of nuclear weapons isn't that they'll be used in large numbers, but that they'll cause a power imbalance that lets nuclear-armed nations extend the nuclear umbrella as a diplomatic tool, act with relative impunity, and use them at a small scale against non-nuclear opponents.
Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems.
Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.
I mean the analogy is especially apt here. The average person literally can't care about a large scale nuclear war. Their truth table has only the options "live normally" and "die".
In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?
> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive.
Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence.
Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will?
Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).
As with most problems in the world, we're being failed by our sluggish, corrupt governments voted for by apathetic, distracted citizens in an endless downward spiralling cycle of more corrupt, more distracted, more corrupt, more distracted.
523 comments
[ 2.7 ms ] story [ 288 ms ] threadAI is looking to be a vehicle for yet another consolidation of wealth, and by proxy, power.
Unfortunately that’s what I suspect too.
While wealth is busy stepping on you with its jackboots it is vying with other wealthy people for resources and power. Because AI is something that will increase the wealthy persons power, there is the counter risk that another persons AI will make them even more wealthy if they don't keep improving their AI.
The wealthy really don't want to get in a very expensive AI war with each other if they can avoid it. They'll seek to regulate it enough they can gain power over most, but setup a regulatory system they feel gives them some form of control.
Anonymity and obfuscation of crime are killers.
AI/ML governed corporations, hedge funds and private equity will be able to be "Enron + Goldman Sachs + Nestle + Monsanto/Bayer + Purdue Pharma" on literal steroids. You might add AGI to mix to attempt to repair the system that was created by the most powerful sociopathic goal seeking super-toddler that the world has ever seen.
I mean, they can be, unless someone already has total power and then they don't give a shit and do crime in the open. Then they demand the lack of anonymity from you and turn the world into a police state.
How do we know this? Because we've had literal centuries worth of learned experience backing it! The exact reason why corporate capitalism is so brutally efficient is because there are lots of things we as humans value but don't have a suitable profit motive for. The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition). But that also runs counter to most AI safety proposals, which usually involve some kind of licensing system, which will ensure a handful of companies have the best and most direct access to the benefits of AI.
Yes, that's exactly the problem, which is why this
> The way you work around this is to decentralize (make sure there are multiple companies competing with one another) and distribute (make sure everyone has some access to the means of production in order to create more competition).
is backwards. The reason competitive markets generally work out better for consumers than monopolistic ones is because they're stronger optimizers, more ruthless, more capable of crushing all competing values underfoot - and those of us lucky enough to benefit from an inefficient, low-crushing segment of the labor market get to collect some of the resulting surplus.
Nothing about this is inevitable: it's just the way we happen to be positioned in one particular system at one particular moment in history. There's no reason to expect that we would remain safely un-crushed in a far more competitive environment.
Have everyone watch War Games and make sure there's a plug.
https://www.youtube.com/watch?v=Jk4T-SxTkWA
"Which political party has had the most politicians convicted of crimes in the last 50 years?"
"According to a comparison of 28 years each of Democratic and Republican administrations from 1961-2016, Republicans scored eighteen times more individuals and entities indicted, thirty-eight times more convictions, and thirty-nine times more individuals who had prison time1. Is there anything else you would like to know?"
Oh wait, that's not bias. The Republican Party demonstrably has a higher number of its members convicted of crime. Simply put, Republican politicians are much more likely to commit crimes than Democrat politicians.
Reality has a well known liberal bias.
Over the past decade social media has nurtured a culture where reality is whatever you choose to believe
I know the language is "choose your reality" and I'm not saying you're wrong, but I think we could acknowledge that's not really what's happening. Folks don't get to decide facts, they decide what facts they believe. That decision doesn't invalidate those facts.
edit: looks like bing pulled it answers from: https://medium.com/rantt/gop-admins-had-38-times-more-crimin...
> According to a comparison of 28 years > 1961-2016
Talking of bias - this is used the world over..
perhaps AI will be generally intelligent, but please for the love of god, let's only consider that scenario what it actually happens.
An LLM is tied to a bunch of APIs as an event response agent. Maybe in some poduck enterprise support group.
But... there's tons of other LLMs tied to event response agents, in ... less podunk groups.
... and there's some fancy black box org in the CIA that then employs them as some crisis monitor.
and a cascade of false reporting leads to "oh my god there's nukes in flight, and the system has autolaunched nukes in response".
What current AI is good at is faking being intelligent, so that they can be used in many many places to substitute for "real intelligence" humans.
But... that means dangerous cascades in all likelihood.
That isn't skynet, but it is 90% of what the Terminator skynet disaster scenario was: an AI triggered a nuclear exchange.
Reason 1: Profit motive: No one's going to be making a profit off of hooking an AI up to nukes.
Reason 2: For the lulz: This is the kind of thing that happens when you have very broad access to the elements in question. Since access to nukes is very tightly restricted, and their usage controlled by significant parts of governments, it's vanishingly unlikely that either a conscious collective decision will be made to hook up an AI to nukes for the hell of it, or that someone will be able to do so on their own.
And yes, I know that there are lost nuclear weapons, which means both that a) there are some out there not under government control, and b) governments can be careless with them—but frankly, any non-governmental actor known to be holding a nuke would have to be considered a top-level threat no matter what they had making the decisions about whether and how to use it.
This only makes sense if you think that at that point, it will not be a "the genie is out of the bottle" scenario and already too late to put it back in. And yet that's what people constantly say about current LLMs.
But you're right in that the two problems are inseparable. Wisely implementing sensible and effective AI controls will fall onto politicians and statespeople, from legislatures to constrain things domestically, to Presidents who have to lead abroad in forging meaningful and effective treaties.
Does AI have to be strong to kill everyone? I can imagine scenarios in which it could be fairly shit, but good enough.
> please for the love of god, let's only consider that scenario what it actually happens.
Not a great philosophy to guide planning.
Let’s hear these scenarios that aren’t possible without AI.
Being more specific will allow for more specific responses. If the danger being addressed is the danger from text-prediction software, call it that. If it's something else, then describe it.
Rather? I'd imagine concerns about autonomous agents running amok is further into the future than the digital content manipulation potential of gen AI possible today. Heck, porn deepfakes of celebs were a problem in 2020, and right now Adobe has made gen AI generally available in Photoshop (AFAIK)
https://xkcd.com/1968/
Misdirection. We see a generic call for regulation, or unrealistic call for a global pause with no answers to how it would be coördinated or enforced. When actual regulations are put forward, they're rejected without a counter-offered solution [1].
[1] https://www.reuters.com/technology/openai-may-leave-eu-if-re...
There is a difference between trying to shape regulation to your advantage and engaging in bad faith. There is no indication there is real regulation that addresses the problem these folks are brining up that they would support.
Many other people talking about x-risk signed the letter calling for a pause on training new big models, which seems pretty concrete to me. Does that make sense to you? If not, do you have other regulation you'd prefer?
It's totally unrealistic. How are you going to coördinate a global pause on AI research?
Nobody has even bothered proposing a framework because of how ludicrous it is for this to be what Beijing, Washington, Brussels and New Delhi focus on.
But it's not clear to me what you think would be a good idea, which makes it difficult to see what you think would be a serious proposal rather than misdirection. It appears to me that you are taking it as evidence of bad faith if someone proposes regulation either stronger (pause) or weaker (OpenAI amendments to proposed EU rules) than what you prefer - but you also haven't said what you do prefer. What am I missing?
Personally, while I don't think the pause is realistic, I don't think advocates necessarily have any obligation to propose realistic ideas when they're not in a position to actually write any laws and are only able to potentially shift the window of what politicians and media consider acceptable.
Registration. Not even licensing. If you're training a model, you file a form describing who you are, who you work for, who you're working with and what you're doing with which data for what purpose. This lets professionals, policy makers and the public get a sense for the landscape so we can make rules with some sense of reality.
Sure. Then the EU proposed something like it and Altman threw a hissy fit. That's where my accusations of bad faith come from.
Bad faith is unnecessary for bad outcomes. The perverse (from our PoV) incentives are sufficient.
OpenAI's incentives is to shut down all the rinky-dink models every hacker and their dog are building on cheap hardware. They spent billions, and are having the market taken from them.
Nobody has proposed any specific and realistic regulations.
The code for AGI will not be some monolith of software architecture. The code will likely be simple. This means that someone in their basement could build it. The steps to get there are challenging, though. A single person could have developed the transformer architecture. A single person could have used 4-bit quantization and developed an AI that is just as good as ChatGPT and have it run on their local machine.
The difficulties are figuring out the best 'needle in the haystack' to solve the problem. This requires research, and this process happens much faster if you have more people working on it. For years, many people did not put any energy into AI systems because the hardware was not here yet. The hardware is here now. The cat is out of the bag.
This just ignores the very real coordination problem. The signatories do not represent the entirety of AI development, nor do they want to unilaterally forgo business opportunities that the next man will exploit. Government is the proper place to coordinate these efforts, and so that is where they appeal.
It smells like bullshit being espoused to push an agenda, but I can't tell what the agenda is. My guess: play up the huge unrealistic risks in order to distract from the more realistic ones.
Two things about this tech that I'm personally not worried at all about: "evil agi" and "only the elites will control this".
There are so many things wrong with this line of thinking. First, it mischaracterizes the issues. Few people believe AGI guarantees the extinction of humanity. The issue is that there is a significant potential for extinction and thus we need a coordinated effort to either manage this risk or prevent its creation. It does little to stop the coming calamity to single-handedly abstain from continuing to build. Coordination is a must. Besides, most people will think they stand a greater chance of building it safely than the next guy. The coordination is required to keep other people from being irresponsible. Human hubris knows no limits.
The other mistake is misjudging nerd psychology. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it". It is a grave error to discount the motivation of trying to solve a really sweet technical problem.
Ultimately these kinds of claims are self-serving, they provide rational cover to justify your predetermined beliefs that those calling for regulation are trying stifle competition. Folks don't want to be left out of the fun. The justification is in service to the motivation.
“Artificial intelligence could lead to extinction, experts warn” - BBC front page news headline earlier today in reaction to this story.
This is the problem. Reading that BBC article will make your average joe petrified. “Extinction” is a much catchier headline than the slow creep of automation replacing/changing jobs. The latter is literally already happening around us right now and it serves some of those signatories if those issues aren’t regulated against [1]. I’m not saying forget about extinction threat. Clearly that’s an important risk to manage, but let’s not ignore these near term, huge disruptions because policy makers are busy reacting to distracting headlines.
Edit: add ref; [1] https://www.reuters.com/technology/openai-may-leave-eu-if-re...
https://en.wikipedia.org/wiki/Don%27t_be_evil
They are grabbing headlines, and moving the conversation from the real issues, which are how AI is used in education, health care, law enforcement, securities and housing markets, the government, the military, and more
https://news.ycombinator.com/item?id=36100525
Long before AI causes mass harm without human involvement, humans will find hundreds of ways to make it cause harm, and harm at scale. I do think the technology itself is part of the risk though, because of the flaws, the scale, etc inherent to its current iterations. However, maybe those are still the fault of humans for not giving it the proper limits, warnings, etc. to mitigate those things
That said, it can be used for good in the right hands (accessibility tools, etc), potentially, though I'm certainly more of a doomer at this point in time.
Why aren't we talking more about it?
Geoff Hinton /left his job/ at Google because he sees the risks as real, so I think that's a tough case to make.
Their message is being amplified and distorted by forces that are. That they're both comfortably wealthy from creating the problems they now preach solutions to is also no small matter.
He did not leave his job when the bias and harm of the existing I models at Google were doing.
He did not quit his job when Google fired their AI ethics champion rather than change their behaviour
Has he had anything to say about the harm Google's algorithms do?
If it were a perfect society and people discovered deep neural networks, there would be no excuse to blame bad actors for posing the question of whether AGI is an existential problem or not, and what to do about it. Unlike bad groups of people, the question won't go away. In the real world, it is entirely possible to have to consider multiple issues at once, not at the expense of any issue.
The incentive here is that they want to live, and they want their loved ones to live.
Perhaps the simplest explanation for why researchers are saying there is a danger is that they genuinely believe it.
A small group of people risking the lives of billions without their consent is morally repugnant.
The other is a hypothetical future scenario that has literally no A->B->C line drawn to it, pushed by people whose ability to feed themselves depends on attention farming.
So no. Let's not focus on the bullshit one, and let's focus on the one that is hurting people in the now.
There is a good real world example of how optimising for a goal aggressively can make the world unliveable for some species, which is humans destroying the environment for our economic growth. But at least there we do care about it and are trying to change it, because, among other reasons, we need the environment to survive ourselves. An AI wouldn’t necessarily need that, so it would treat us more like we treat bacteria or something, only keeping us around if necessary or if we don’t get in the way (and we will get in the way)
It might sound silly but there is some solid reasoning behind it
> literally no A->B->C line drawn to it
Spend a couple of hours to educate yourself about the issue, and make a convincing counterargument please:
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
https://astralcodexten.substack.com/p/why-i-am-not-as-much-o...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
https://www.youtube.com/@RobertMilesAI/videos
https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintel...
> For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again.
I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)
Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.
These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic.
It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.
"A God is a mind that is much more intelligent than any human."
"Go programs had gone from “dumber than children” to “God” in eighteen years, and “from never won a professional game” to “Godlike play” in six months."
"God has an advantage that a human fighting a pack of lions doesn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically."
"A God might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with."
"Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical God is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways."
(Oh, god, there's my personal bugbear: "bigger brain size correlates with greater intelligence". You know what else correlates with bigger brain size? Hint: I'm 6'5"; how much smarter am I than someone 5'1"?)
Are you suggesting human intelligence is unique and the upper limit for intelligence? I mean we can argue about the difficulty and timeline for achieving human level artificial intelligence, but it seems unlikely that it's impossible.
So, if you're not worried about the extinction risk of God, why should you be concerned with the risk of "superintelligence"?
It's more of an inevitable consequence (over an unknown timeline, granted) of the assumption that human intelligence is not some kind of absolute limit, and can be reproduced in a non-biological system. I admit that is still an assumption, but the opposite seems harder to support to me, though we're all admittedly guessing here.
Once you make that assumption, then yes of course it follows that the 'beyond human intelligence' would be able to do all of the things that human intelligence is already doing. Namely, science, mathematics, developing AI systems etc.
Sure, the probability of AI-triggered extinction is lowish, but the consequences are essentially infinite, so we are justified in focusing entirely on the AI-extinction threat. (But I guess it's not infinite enough to justify rounding up all the AI researchers and shooting them?)
It's rhetorical sleight of hand.
While putting an AI in charge of weapons is at least three Hollywood plots[0], it has also (GOFAI is still AI) behind at least two real-life near-misses from triggering global thermonuclear war[1].
[0] Terminator (and the copycat called X-Men Days of Future Past); Colossus the Forbin Project; WarGames
[1] Stanislav Petrov incident, 26 September 1983; Thule Site J incident October 5, 1960 (AKA "we forgot to tell it that the moon doesn't have an IFF transponder and that's fine")
It seems to me they're inconsistent ways of understanding what is happening, since concern about misuse also seems to me to imply regulation. But I seem to see a lot of people on HN who say both things who give the impression that they're agreeing with each other.
MS Azure Cloud is. Anthropic (very alignment/safety-centered company) is backed by Google and their cloud. Any API tokens sold are just cloud credits with markup.
So to me the doomerism does come from those entities acting as fronts to the major players that would very much like to build a regulatory trench.
Setting aside the plausibility of this degree of coordination, I don't even see the alignment of interests. What does Google or Azure care of whether they sell GPUs to quasi-monopoly or a bunch of smaller competitors? In fact, isn't the standard economic argument that monopolies underproduce, so competition should if anything sell more GPUs? Meanwhile, promoting doomerism seems pretty risky for the GPU business - they have to promote it just enough to get hype and regulatory capture, but not enough that their ability to buy and use GPUs is actually restricted. Seems like a risky bet for a corporate exec to make, unless they can really control the doomers with puppet-like precision...
what gets published by academia is based entirely on who gets funded
my conspiracy theory is a stretch, granted, but to clarify:
It behooves the cloud providers for their customers to believe that their latest $1/hour l/user upcharge is revolutionary (so much so that ethicists are shouting please, stop, you know not what you do!)
OpenAI and Anthropic need the public to trust them and only them as providers of "safe" AI (not like the other open-source AI that might turn into an especially persuasive holocaust denier any minute) - so from the regulatory angle they want to play up the theoretical dangers of AI while assuring the public the technology is in good hands.
As for the academics, well it's not like anyone gets funding for writing boring papers about how nothing is going to happen and everything is fine. No one has to be puppeteered, they just need to look around and see what hype train might direct some funding their way.
I read the article you linked to, both parts. I wonder how much of people having psychotic breaks in the rationalist community is due to 1) people with tendencies toward mental illness gravitating toward rationalism or 2) rationalism being a way to avoid built-in biases in human thought, but those biases being important to keeping us sane on an individual level. (If you fully grasp the idea that everyone might die, and have an emotional reaction to that that's proportional compared to just one person you know, it can be devastating). I think we are bad at thinking about big numbers and risks, because being very good at evaluating risks is actually not great for short term, individual survival -- even if it's good for survival as a species.
I know personally the whole AI/AGI thing has got me really down. It's a struggle to reconcile it with how little a lot of people seem to put stock in the idea of AGI ending up in control of humanity at some point. I totally agree that everything on your list is a real issue -- but I think that, even if we completely solve all those issues, how do we not end up with a society where most important decisions are eventually made by AGI? My assumptions are 1) that we eventually make AGI which is just superior to humans in terms of making decisions and planning, and 2) there will be significant pressure from capitalism and competition among governments to use AGI over people once that's the case. Similar to how automation has almost always won out over hand-production so far for manufactured goods.
That's more the scenario Paul Christiano worries about than Yudkowsky. It seems more likely to me. But I still think that a lot of our mental heuristics about what we should worry about break down when it comes to the creation of something that out-guns us in terms of brainpower, and I think Yudkowsky makes a lot of good points about how we tend to shy away from facing reality when reality could be dreadful. That it's really easy to have a mental block about inventing something that makes humanity go extinct, where if it's possible to do that, there's no outside force that will swoop in and stop us like a parent stopping a child from falling into a river. If this is a real danger, we have to identify it in advance and take action to prevent it, even while there are a bunch of other problems to deal with, and even while a lot of people don't believe in it, and even while there's a lot of money to be made in the meantime but each bit takes us closer to a bad outcome. Reality isn't necessarily fair, we could be really screwed, and have all the problems you mentioned in addition to the risk of AGI killing us all (either right away or taking over and just gradually using up all the resources we need to live like we've done to so many species).
https://aiascendant.substack.com/p/extropias-children-chapte... (and I still recommend all 7 parts)
It's hard for me to respond to the rest your comments, because I simply don't agree with the framing. To me I see a big trap where people read a lot of words, but don't connect those words to reality, which requires action.
If you only manipulate words (e.g. "intelligence" and "AGI" are big ones), then you're prone to believing in and reflecting illusions. He was upset that I wasn't upset.
---
You didn't ask for advice, but you say you're not feeling good. I would draw an analogy to a friend who in 2017 was preoccupied with Trump possibly causing a nuclear war with North Korea. There were a lot of memes in the news about this.
Now certainly I couldn't say at the time that it was impossible. But looking back 6 years, I'm glad that I completely ignored it and just lived my life. And that isn't to say it's gone, and not real -- certainly something like that could happen in the future, given the volatile political situation. But it's simply out of my control. (And that's what I told my friend -- are you going to DO anything about it?)
Regardless of what you believe, I think if you just stop reading memes, and do other stuff instead, you won't regret it in 6 years.
I agree with the Substack in that rationalists can be curiously irrational when it comes to examining where they get their own beliefs. (Or maybe examining them in a "logical" way, but completely missing the obvious, common-sense fact -- like MacAskill writing an entire book about moral actions, while his life's work was funded by a criminal -- a close associate who he vouched for!)
Like EVERYONE ELSE, they are getting beliefs from their reptile brain ("world will definitely end"), and then they are CONFABULATING logical arguments on top to justify what the reptile brain wants.
And you can't really change your reptile brain's mind by "thinking about it" -- it responds more to actions and experiences than the neocortex. It definitely responds to social status and having friends, which can lead to cult-like behavior.
I'd say that humans have a "circuit" where they tend to believe the "average" of what people around them believe. So if you read a lot of "rationalist" stuff, you're going to believe it, even if's not grounded in experience, and there are gaping holes in the logic.
---
The big trick I see a lot of "rationalist" writing is the same one used in the Bostrom book.
1. You state your premises in a page or so. Most of the premises are reasonable, but maybe one is vague or slightly wrong. You admit doubt. You come across as reasonable.
2. You spend 300 pages extrapolating wildly from those premises, creating entire worlds. You invent really fun ideas, like talking about a trilling trillion people, a trillion years into the future, across galaxies, and what they want.
You use one of those slightly vague premises, or maybe a play on a word like "intelligence", to make these extrapolations seem like logical deductions.
You overthink it. Other people who like overthinking like what you wrote.
You lack the sense of proportion that people who have to act must acquire -- people with skin the game.
Following this methodology is a good way to lead to self-contained thoughts, divorced from reality. The thoughts feed on themselves both individually and socially. If you're wrong, and you frequently are, you can just draw attention to some new fun ideas.
---
Anyway, glad you have followed the project. Speaking from experience, I would suggest concretely to replace the rumination t...
> Do the signatories believe that existing AI systems or their immediate successors might wipe us all out? If they do, then the industry leaders signing this statement should immediately shut down their data centres and hand everything over to national governments. The researchers should stop trying to make existing AI systems safe, and instead call for their elimination.
But a different set of people called for a halt to AI research two months ago, and his response then (https://aisnakeoil.substack.com/p/a-misleading-open-letter-a...) was that these concerns are "fever dreams" and we should disregard them in favor of "serious policy debates".
It seems like there's a self-reinforcing system here, where any evidence or argument in favor of existential risks from AI can be dismissed out of hand.
He is saying "if those people genuinely believed what they were saying, they should logically shut down their own AI systems."
His position is consistent and clear: the supposed threat from AI is based in sci-fi and motivated reasoning. Indeed, even those who are motivated to spread the idea of that threat do not appear to genuinely believe it—either that, or they believe that their own right to continue to do whatever they can to maximize their profits outweighs it.
- Jon Lajoie
This is where you lose me. AGI (not LLMs, but intelligence smarter than us) is qualitatively different from anything else we've ever made. The one example we have of this in history is the development of human intelligence. It did not work out well for other species, and now we own the Earth. This isn't a perfect analogy! There are some reasons why AGI may be better (we'll be actively trying to align it to human interests) but also why it may be worse (there will be pressure to make it more powerful and take off guardrails).
There may be good reasons for why we'll be able to control AGIs, but "history of technology suggests..." is not one of them.
And I'm not arguing that all the "human misuse" problems are not important! They're hugely important! A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on. I don't object to working on the former problem, except to the degree that the solution makes the latter problem worse (like, "open source all the models", which is likely to happen at some point anyway).
Rich getting richer [x]
Poor starving [x]
War [x]
Powerful gain more power [x]
Based on how I already see the world, honestly, I'm not sure that's much of a prediction.
This will get worse of course through LLM-powered fake news barrage funded by powers interested in destabilization, but that's bad enough already without AI anywhere near. From this angle, LLM are more like the introduction of a more powerful gun than like the invention of gunpowder.
I look forward to a "day the earth stood still"-like overlord that punishes aggression, greed and corruption: current human systems seem to be inadequate at solving these issues.
If you mean AGI in the sense of singularity, where it can self improve and get smatter, the question of that even being able to happen in the first place relies on P=NP being proven first.
So it's not like humans controlling other humans, it's like chimps controlling humans. It might last a while -- maybe a long while. But it doesn't seem stable.
Lets say we have a super smart AGI and with more memory than a human, no emotional faults, and give it access to internet, as well as manufacturing facilities where it can make robots and copy itself onto said robots.
The issue in this AGI taking over the world is that it would first have to gain control of a the sole resource necessary to its survival, which is electricity, which it won't be able to do easily if we detect it going rogue.
For an AGI to be so advanced past singularity as to parse and compute on reality to know exactly the steps it needs to take to secure power source, (or in the more common example, successfully break out of the box) would involve it essentially being able to compress the computationally irreducible processes that happen in our world down to an equation, which would mean that P=NP.
But it really can't today, not without a bad actor giving it specific goals and figuring out ways to make those yield real-world outcomes. Which is the entire point of the article.
Unless an oppressive adversary has such weapons (ICBMs, antipersonnel mines, or machine guns), wants control of your land, and decides to take it. In that situation, a country will seek to utilize such or more powerful weaponry for themselves to defend their people. If that country (1) did not manufacture such weaponry inside their borders and (2) were prevented from purchasing such weaponry from other countries by blockade, sanction, or other means, then yes, those people will be oppressed.
Joke's on you, because I can absolutely buy one tomorrow. Thankfully you don't make the laws :D
AI risk is a valid discussion but regulating AI in its infancy seems misguided at best.
Misinformation has been a problem for all of human history, in its current form dating back to at least the Spanish civil war.
All the existing version of AI does is replace troll farms.
This isn’t a new threat, it’s a faster bullet.
Well, now we have $500 drones with an RPG strapped to it, and you CAN task a guided munition to each and every soldier and piece of equipment. There is no safe space in the field. You have to dig holes in the ground and hide from the sky. The lethality of war has gone way up.
These "AI" models are the same way. Before, it was expensive and effort consuming to run a scam or phishing or anything that requires social engineering. That's why the target the soft and valuable targets, like grandma, or people on MLM mailing lists, or large and juicy companies. But now, the cost of a phishing or scam campaign can be heavily reduced. Think of how often companies fail internal phishing tests, and those emails are usually pretty basic, low effort, trivially identifiable as phishing. It's a massive upgrade to troll farms. They don't have to limit their targeting as much, because cracking harder targets will be cheaper to attempt. It doesn't matter if it isn't actually more efficient or more effective, just the fact that targeting someone and initiating a campaign against them is cheaper is all that was needed to make this situation brutal.
I would argue you're wrong, because there is a long, colorful history of rulers oppressing populations by making the possession of weapons illegal. If the populace has no means to fight then they can't revolt. I'm not saying my neighbor should be able to buy ICBMs, but it's definitely true there's some element, no matter how small, that he's being oppressed by not having access to the same level of force as his rulers (the government).
For instance, what happens if one company gets to control the future of social media, education, etc. through AI? That company can then unilaterally dictate our minds.
That risk doesn't map to the gun thing at all.
The difference between guns and automobiles is in the name. Cars are mobile by themselves. That’s a slippery slope waiting to happen.
It’s been called “AI” ever since they were puny inference programs. Them having “intelligence” in the name signifies nothing. Only what they can do in reality.
Who here with experience does not think these neutered late releases do not tilt the playing field?
"Open"AI was founded assuming that premise. Talk about a 180.
One also can't replace nurses with guns to make healthcare more scalable.
There's a mob of dreamy libertarian techbros on HN who are (often wilfully) ignorant of the dangers. I'll start worrying about military robots after we figure out how to prevent the Earth from being dismantled, which very well might transpire first, if and when there's significant architecture breakthroughs.
AI does not need to be evil, it does not need to be conscious, it does not even need to be severely misaligned with human values, and it really doesn't matter if the goal of "eat all the humans" arises from a model hallucination or from a misanthropic troll in his basement typing it into his looped uncensored-LLaMAv3+plugins instance.
I submit that it is not anything special, certainly pales in comparison to the atomic bomb. It's merely that we live in that weird end tail of the age of ideology where strong overarching ideologies came to (and are coming to) an end, where we're all scattered to the various geopolitical winds and in the absence of a strong guiding state and vision (communism vs capitalism) become more extreme and perturbed.
If you do want regulation, It should be something proper hashed out diplomatically and economically between Nation States and their respective blocs, no crony capitalist allowed, sorry. The rest of the world shan't be going along with something that economically cripples them.
It's only after decades of scifi and Hollywood movies that people thought, "oh, we could accidentally create a bomb that wants to kill us all".
Both are true.
People are jumping too quickly to deeply integrate this tech with everyday things, and while that's great for many use cases, it's not so great for others.
You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.
What data would comprise such a model?
They provided an example and finding a technical flaw in the example they chose doesn't invalidate the broader concern as applied to other domains.
"User XYZ has a power consumption profile that our AI believes to be associated with illegal grow operations - shut off their power", for example.
It's the shutting off power that's a problem, not the AI.
You'll catch shit for taking direct action yourself. But if your AI does it, without you telling it to..."oops!"
Tech is dead. Welcome to the age of social engineering.
"""Here is a person's power usage history:
{{ power usage by month }}
Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any:
{{ history }}
Here is their credit history for the last five years:
{{ credit history }}
Here is the location of their home and some overall information about the grid:
{{ grid info }}
Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing".
"""
Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles."
If you imagine a less regulated industry there's even more room for price discrimination and such.
The issues, I think, are at least three-fold:
1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too?
2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do.
3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.
The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics.
This sounds very far fetched to me.
Would it surprise you if your utility pulled your credit score? Cause that already happens.
Why is it far-fetched that they wouldn't take another step or two along that path?
And let's say it isn't the power company. Let's say it's your employer. Feel good about that?
Regulations would prevent this type of abuse to power consumers in the United States. The rest of this argument is whataboutism.
So... you agree with the original post now that there's a specific risk with the current potential uses of these models?
I'm not sure what you're questioning at this point.
("what if it was your employer" is also not "whatabouttism," it's another facet of the same concern. Another party who pulls publicly available data today, and could potentially pull even more, giving us as the public the need to decide if we want to allow that to happen. Should someone be able to be made unemployable if an LLM decides they're too much of an asshole on Twitter? Let's figure it out.)
No. I do not believe the US government would allow public utilities to set prices based on social metrics bought from 3rd parties.
>"what if it was your employer" is also not "whatabouttism,"
I do not believe a public utility and my employer correlate. Maybe if I was a government employee. Even then, I don't believe the government would have the right to use that data against me either.
I have a hard time believing these underhanded tactics are overlooked and allowed.
Imagine you'd never read the original comment and instead you read a comment from me, saying, "I have a specific concern about how LLMs like ChatGPT will let companies do far more intrusive background checks against every applicant than they do today. I don't want a world where the standard process for getting a new job includes a background check that runs an LLM across everything you've posted on the internet. It wasn't practical for them to do this for every single applicant in the old world, they just did cursory background checks since doing more would cost too much, but as LLMs get cheaper it will be easy and cheap."
Do you agree that that's a concern introduced by the development of LLMs?
Really, we're looking at the wrong place talking about the power company... where you should be looking is rental properties.
https://www.propublica.org/article/yieldstar-rent-increase-r...
When I worked at a utility company, we had a purchased dataset on customers with things like inane, ridiculously specific features such as "average number of game consoles per household", on street level.
Now,
a. the data quality was absolute dog shit, and
b. the idiotic amount of features made it super easy to overfit, leak or otherwise do untoward shit when training a model, and
c. often the resulting models (churn damping, targeted marketing) didn't perform significantly better than random sampling...
But the business users/POs ate up that shit like fine cuisine italian sandwiches, because "we have comprehensive 360' multichannel whateverthefuck" insight into our customer base and we make meaningful business decisions based on this.
And this is in GDPR-crazy europe.
I see LLMs rapidly increasing the data quality of those sorts of datasets by enabling full-text crawling of all sorts of other publicaly-available or purchasable streams of text. You said on HN once that you had 5 consoles? Well, we matched your hn username to your username on this other site, and there was a breach that matched usernames to emails over there, and there was a different breach that let us match emails to full names, and bam, now we have an accurate number.
No, now we THINK we have an accurate number because everyone in the entire chain is bullshitting about how good machine learning and their data is.
That's so much worse.
Planning coercive disconnections in order to maximize profit from a customer based on credit history seems like the problematic thing in this example. Asking an LLM for recommendations on doing seems unrelated.
If they asked a human consultant for it instead, it would be just as bad. And just because the human's recommendations would be more explainable, it doesn't make the human consultant's recommendations any less problematic.
Asking a human consultant to take a human-intelligence-using look at every single customer (or every single visitor to a website, or whatever) isn't realistic at scale. A company will pay a bit for a background check for a prospective employee, they won't hire a PI to read every single piece of social media correspondence that person has ever written.
So if LLMs get cheaper and cheaper, there's a major scale difference due to the existence of the tool.
Technological developments leading to re-thinking laws and regulations is nothing new, of course, and "we should outlaw employers hiring PIs to read every piece of text a human has ever posted online" is something that never would've been a realistic thing to pitch to a politician 25 years ago.
Now, the technology has changed, so we need to talk about it.
You seem to be agreeing that it would be bad, so like the original poster here proposed, let's focus on figuring out what "bad" use cases this tech makes far cheaper and more realistic today and then trying to convince others that we should regulate those use cases.
(The second thing I brought up is a difference not in scale but in deniability and ass-covering: it's illegal already to take a retaliatory action against an employee who complains about harassment, for instance. Figure out a sneaky-but-very-plausible way to make that employee just-happen-to-come-up-bad in some LLM-mediated part of a performance review process? Now you'd have a much harder time getting caught.)
Taking off my Republican hat: which is why up-front regulation on specific actions and methods is what we need instead. Much easier for other people to spot, much more of a bright line "you did this thing, we said don't do this thing." Not at the granularity of "don't use ChatGPT specifically" but more like "these are the things we will allow and won't allow in how you process job application background checks" (we already do this for discrimination, I just think we should update it to reflect that we don't want a centralization/standardization of process so that people become de facto unemployable based on what some tool used by some company thinks about them).
[0] not to mention that looking for bias in outputs in a mechanical way is also gonna false-positive a bunch; p-hacking, but for accidentally getting sent to jail :(
LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.
That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).
It's going to be a dark day when some idiot passes a law forcing AIs to serve as mandatory reporters.
This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic.
You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.
Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
This is very much not what they do.
I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes.
My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI.
I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content.
Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work.
I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job.
I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy.
It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI.
Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.
Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.
This smacks of misinformation. If you're referring to that dreaded basilisk, very few people actually believe that scenario has any likelihood at all. Also, it has nothing to do with time travel anyway.
The other two things are more or less guaranteed because they're simple extensions of processes that are happening right now and can be seen all around us.
Since he's become, for many, the "face of AI risk", that gives it much more (undeserved) appearance of credibility than you imply.
(And apologies if I misled people into believing the superintelligence basilisk has time travel powers)
I don't think that's true; he's stated before he doesn't and didn't believe it was true. His reaction was more about the stupidity of posting information hazards on public fora rather than that specific information hazard being credible. However, he could be lying post-facto.
And I do agree that it kind of sucks that Yud is the "face of AI risk" because it doesn't even matter how correct he might be; a lot of people just don't like him (I'm indifferent). There shouldn't be a "Face of AI risk" because there won't be a single person that everyone like. The idea is bigger than him.
Or weird nanobot bullshit.
Is there a really boring possible outcome that leads to the destruction of humanity?
The second biggest risk is just humans being afraid. A scared human is a dangerous, irrational human.
It doesn’t have to be boring, but boring sounds more likely to me, because it’s soemthing we don’t think much about… because it’s boring.
I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then.
I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions:
I remember one of the first conversations I had on "global warming" specifically and it was around 1996. But even before I had heard that specific term we were talking about losing large coastal regions due to rising sea levels. The rock band Tool released the album Ænema in 1996 and there is the constant refrain: "Learn to Swim". By the time Al Gore published "An Inconvenient Truth" in 2006, I was feeling the topic was a bit played out, to be honest.At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do.
1. https://en.wikipedia.org/wiki/Paris_Agreement
Even boomers were taking the risks of climate change very seriously.
There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier.
Although I was a child then, I even remember a lot of serious agitation happening in the 70s.
The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1]
We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900.
"Some people didn't understand the science, and so called it fiction" isn't a very strong argument.
[0] https://en.wikipedia.org/wiki/Wright_brothers#Flights
[1] https://en.wikipedia.org/wiki/Bernoulli%27s_principle
No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.
Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent.
Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?
What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.
It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility.
> Anyone who was working with transformers could have seen ChatGPT on the horizon
So, who did?
> [LeCun is] highly critical
I did a quick search and found this[1] trash-tier article explaining his views summed as:
> He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial.
I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth.
Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem.
Finally, your question:
> What is so impressive about ChatGPT that it poses an existential threat?
This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief:
> Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals.
So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow.
The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
> essentially 0%.
I sure hope you're right, but I don't think that's a common position to hold.
[1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...
> The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
It’s not assuming it won’t happen or burying our heads in the sands, it’s calling for measured responses.
My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
Autonomous AGI, whenever that happens, absolutely poses a threat. But it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
People assume advancements will continue to happen at their current pace. The last 6 years of advancements has been the product of scaling and refining self-attention + transformers from 2017/8.
> so who did
I’m not sure what you’re getting at. No one had the will/resources to burn billions on a language model that has ??? business plan, doesn’t mean no one saw it coming.
I can appreciate that. I haven't been listening to him for that long; having no idea what situation you're discussing, I'll take it at face value.
> My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
That's pretty reasonable. However, I will reiterate that this:
> it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
is not what I'm asserting. It's the unknowns which are dangerous. We're inventing something new in almost every way imaginable. It's not like we're talking about a new theoretical steelmaking process which could upset the economy, we're talking about the potential to build a new life form with godlike abilities. Alarm seems warranted to me. Sure, it might not happen soon... but also, it might happen soon. Shouldn't we be trying to prevent it, or control it, before it happens?
> People assume advancements will continue to happen at their current pace.
I think people assume advancements will accelerate, actually.
https://m.youtube.com/watch?v=2HMPRXstSvQ
We are currently facing the biggest radiologist shortage in the last 30 years.
> It’s the unknowns which are dangerous.
Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.
> We’re inventing something new in almost every way imaginable.
> I think people assume advancements will accelerate, actually.
This is highly debatable.
> we're talking about the potential to build a new life form with godlike abilities.
Evidence? This has been stated in sci-fi books from before I was born and I don’t see any proof that we’re building something remotely close to this.
In a way, all of those things have led to where we are now. And might I point out that a form of bioengineering may have caused the latest global pandemic? Not taking these things seriously because "they haven't killed us all yet" seems a little shortsighted.
> And might I point out that a form of bioengineering may have caused the latest global pandemic?
Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?
Would that have prevented a nation-state from potentially causing a pandemic?
I mean, we should have better regulated carbon emissions generations ago. We'd have a lot more time to deal with it, because we haven't been able to deal with it in the century+ we've known about it. And AI extinction risk is likely to move faster than carbon-related climate change.
I don't pretend to have the answer of what the balance is for oversight and regulation to optimize safety and innovation - but the first step for those responsible is to recognize the potential dangers and unknowns, and produce plans that can be discussed and debated.
> Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?
How about gain-of-function research? How about nuclear weapons? I lean libertarian but also recognize we have a responsibility to safeguard humanity.
Did they? I've never heard about electricity creating fears about human extinction.
The industrial revolution does pose an existential risk: climate change.
Did bioengineering garner worries about existential risk? It certainly had and has a lot of people worried about various risks. And then lots of countries banned human cloning: https://en.wikipedia.org/wiki/Human_cloning#Current_law
> Evidence? This has been stated in sci-fi books from before I was born
Yes, many sci-fi authors of the past were very forward-thinking and predicted future technologies... that's sort of the point.
> and I don’t see any proof that we’re building something remotely close to this
Again, this is a fundamental misunderstanding of the risk.
1. We are building AI. AI is possible.
2. We are advancing AI. AI advancement is possible.
3. We have not built superhuman AGI. But superhuman AGI is probably possible.
The fact that AGI doesn't exist today is frequently argued as a reason we won't have it in the near future, but it's a non-sequitur. We will definitely have more advanced AI in 5 years than we do today, and we can't say what that AI will be capable of. Therefore, it's possible it will be AGI.
And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.
It’s true that once we understood the concrete mechanisms and could evaluate alternatives, not mitigating the risk has been irresponsible, but that was…significantly later than the Industrial Revolution itself.
Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change or any other existential threat. It might have made us more vulnerable to natural pandemics... but it also would have slowed the emergence of natural pandemics and virtually prohibited artificial pandemics.
Neither do I, but then, 100 years ago doesn't clearly meet the description, anyhow. The discovery of the greenhouse effect and CO2’s role in it was after not very long after the dates usually used for the end of the (first) industrial revolution (which is usually what is meant without modifiers, not the second—fourth industrial revolutions) and well over 100 years ago.
Replace "100 years" with whatever timeframe is perfect for you and I'll make the same argument. Please respond to that instead of nitpicking.
At any time before we had the information to understand thr mechanism of problems and evaluate mitigations, its very unlikely that we would choose tbe right mitigations. Sure, if we choose thr mitigations that we would choose based on information today before we had that information, that would be great, but its at least as likely that any attempt to mitigate risk of unknown mechanism vefore we had that infornation would have done exactly the wrong thing.
Basically, “Well, I have information today by which I can design a policy which would have been beneficial if implemented before that information was available” is not an argument that people seeking to mitigate a vaguely imagined risk without any understanding of its mechanism would have the means to design a productive intervention.
I'm saying that with what we know now, it's clear we could have made different choices that would have worked out better for us earlier in industry.
You're saying that the information at the time made it impossible to correctly make the optimal choice. (By that I mean, making the optimal choice would have appeared to be the wrong choice, based on information available.)
I don't agree with your point; I think we knew that burning carbon was going to change the atmosphere for the worse and we could have been more careful by simply burning less carbon. However, I concede that it's unknowable what science and technology would look like if we had done that.
In analogy to AI, that is very much a point of conversation in alignment/safety discussions. Should we wait until we have better tools to figure out alignment, since it's so insanely difficult to figure out now? Frankly, nobody knows for sure the answer to that question either. My contention is that you (and others) are assuming the answer to that question is yes, but that's not necessarily correct, even based on all the information and analogies we have now.
"Dumb" AGI is yet to be proven theoretically possible let alone superhuman.
You can conjure fantastic (positive or negative) unknowns about anything. Unless, for the negative case, you have a concrete reason for believing some class of net-adverse events are probable along some particular route and some concrete basis for evaluating whether alternative actions will both mitigate that risk and not create greater new ones, you have no rational basis for acting based on those unknowns.
It's not something conjured, though. Unless you believe superhuman AGI which can set its own goals is impossible, or that it's possible but it can't possibly have goals that pose a threat to humanity, then it's reasonable to believe we can invent superhuman AGI which threatens our own existence. Technologically, I think it's possible to do so.
So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible? I think we are. Even though we're mostly focused on LLMs right now, and even if LLMs are not a path to AGI (which I agree they probably aren't on their own), it's not like there's no research happening on other pathways. Additionally, we can't be sure that current LLM research won't contribute to AGI - maybe the real path is a synthesis of multiple forms of AI.
So there are some very specific unknowns standing between us and possible doom. We're not making up fantasies wholesale.
- Dangerous AI is possible - We're racing to create it as soon as we can - We don't know how to tell if we've crossed a threshold into having dangerous AI
That last point is an known unknown which is extraordinarily dangerous. It's like if you're blindfolded and moving your hand toward a running chainsaw. You know that if you move your hand far enough it will be shredded, you know you're moving your hand forward, you just don't how close your hand is to the chainsaw. You should be concerned in that situation.
It is something believed without any evidentiary basis which enables rationally weighing the actual risk or evaluating mitigation strategies so as to weigh their probable benefits against their probable costs and the associated changed in the overall risk profile; I'm not suggesting literal magical incantations when I say “conjured”.
> Unless you believe superhuman AGI which can set its own goals is impossible,
Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not. But I don’t actually know that this is important, or even that the mucj weaker assumption that it’s goals might not exactly represent human intent (which is definitely possible of AGI itself is) is important, since human intention can itself be an existential risk.
> or that it's possible but it can't possibly have goals that pose a threat to humanity
Its definitely possible for an intelligence, even a human one, to have goals that are a threat, including an existential one, to humanity. That’s an existing x-risk without superhuman artificial intelligence. It may even be mitigated by superhuman artificial intelligence (as might other x-risks like “earth gets smashed by an asteroid”), if it is possible for such to exist without goals that are antithetical to human survival, in which case delaying superhuman AI could increase existential risk.
But, while this is all nice and conceivable, we have no way of balancing the net impact, positive or negative, of any course on AI to existential risk due to friendly or hostile superintelligences.
> So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible?
No, I can't agree with that, as it would require that every resource that could advance superhuman AI, and specifically dangerous* superhuman AI, was being directed as effectively as possible to that end. I don’t think we are anywhere close to that. Because AI has some mindshare beyond immediate economic utility, we’re probably heading toward AGI, if it is possible and if our current understanding of AI isn’t so wrong as to make our work irrrlevant even in eliminating bad routes, slightly faster than immediately-apparent economic incentives would promote, but nowhere close to as rapidly as possible.
But that's exactly true of either position you take. Imagine I'm arguing that aliens probably exist and you're arguing they probably don't. There is no such thing as a belief based on evidence in this case; we only have logical reasoning and assumptions and beliefs.
> Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not.
This feels like moving the goal posts... all it needs to be able to do is make decisions that are dangerous to humanity. I felt that definition was obvious. All it needs to do do is mimic an oil executive, because, as you say, human intent is also an existential risk. Imagine an oil executive that could get the upper hand in all possible negotiations and desired to make the entire global economy subservient to the needs of oil production.
> delaying superhuman AI could increase existential risk.
Oh, yeah I totally agree that we can't truly know one way or the other. But it kind of feels like deciding to invent and deploy nuclear bombs as a deterrent to war. Did it end war? Absolutely not. Many would argue it didn't even that specific war. We are of course fortunate that nuclear war has not yet destroyed civilization, but a) there's still time, b) we aren't necessarily better off than if we had no nukes, and c) not all risks are equal - AI could be more dangerous.
> as it would require that every resource that could advance superhuman AI
Actually, you're right. I was definitely overstating our progress and didn't think through that argument clearly. But maybe it would be safer if we were devoting all global resources toward a coordinated AGI alignment and development effort...
I think that, specifically, is what is not in the cards.
Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.
I expect that it's possible for superhuman intelligence to exist. I don't think I need to prove that it's physically possible for a superhuman intelligence, natural or otherwise, to exist. Do you think it's not possible? If so, I'd say you need to prove why you think human-level intelligence is literally the pinnacle of intelligence that is physically possible in the universe (especially when narrow AI are already superhuman at many tasks).
Artificial intelligence exists and can be created by humans. This could be disputed if you wanted to gerrymander some really specific borders around what "intelligence" is, but if we don't agree that artificial intelligence exists today, just say than and we'll be done talking.
AGI does not exist. I assert that it can exist, because narrow AI can be improved. Do you believe it's not physically possible for human-level AGI to ever exist? If so, you need to prove to me why we can create narrow AI that is superhuman but cannot ever synthesize them into a human-like intelligence. (And also please give me the scale by which you're measuring an alien brain against ours.)
And if we both agree on all the above, then I take my assertion further: It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI (in fact, I think that if it's possible to create AGI which matches humans exactly, it's probably easier to surpass human levels than to match them exactly, because it's hard enough to measure human intelligence).
So if it's physically possible for us to create superhuman AGI, what exactly am I proving? Do I need to invent the technology in order to prove that the concept is possible?
Of course.
> I expect that it's possible for superhuman intelligence to exist.
I agree.
> Artificial intelligence exists and can be created by humans.
This is where we disagree, although as you say, it depends on what you mean by "intelligence". I think in the context of this discussion, we're talking about some sort of superhuman intelligence with its own will and perhaps consciousness.
In the largest possible sense, it's probable that creating such a thing can be done. Where we probably disagree is that I think we're a very, very long way from being able to do that.
Nothing we have now makes me suspect that we're anywhere near that sort of thing.
> It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI
Agreed. But I assert that we're nowhere near being able to create human-level AGI right now.
> in fact, I think that if it's possible to create AGI which matches humans exactly
I think this isn't possible at all unless you also create an entire human body.
> what exactly am I proving?
The assertion that AGI is a thing that is likely to happen soon. Personally, I think it's vanishingly unlikely to happen within our lifetimes, and very unlikely within the lifetime of our children.
Freaking out about the risks of AGI right now is a serious overreaction. Calmly pondering it would be more appropriate.
So the fundamental disagreement is over how close we are to dangerous AI.
If you're right, we might delay useful AI by decades or centuries being overly cautious. Useful AI could either alleviate untold suffering, or it could cause untold suffering and inequality by misuse.
If I'm right, we could go extinct for lack of caution.
And given that we really don't have any way of knowing with any amount of certainty how close we are to AGI, you see why people are concerned, right? Calmly pondering feels like a serious underreaction. Vigorous, frequent, and open debate would be more appropriate. I'm not saying let's turn off all our computers for a decade.
When we improve LLMs, won't they just be a fancier auto-complete?
They're not a "new life form with godlike abilities".
They sure are fooling a lot of people though.
Not necessarily, and if so, only if your first postulate is true. Which is really difficult to say because "fancy auto-complete" is too vague to argue. Human speech can also be described as fancy auto-complete.
Luckily we don't have to argue that because I specifically am saying it doesn't have to be LLMs. You've missed the point entirely.
Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.
You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
I do think that that level of shortsightedness is absolutely likely in any industry, any company.
Now that you say it that way....
...Enron....
GPT4 has been a great teacher, I love it, but I think a lot of people and industries are already "faking it" and they'll be happy to turn any decision over to AI as long as it makes them look good in the short term. We won't see this from the outside either, like the lawyer who let GPT secretly do his work, others will do the same, but won't get caught like the lawyer did.
https://developer.nvidia.com/blog/exelon-uses-synthetic-data...
https://stanton.house.gov/2022/9/house-passes-stanton-s-dron...
So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?
Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs.
I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to:
> I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning:
1. "LLMs are an amazing technology [which advanced at an unexpected rate]"
2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research
3. Such acceleration could conceivably happen or be happening now in other forms of AI
4. The path to AGI is COMPLETELY unknown.
5. The origins and structures of consciousness and agency in intelligent systems are completely unknown
6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life.
7. AGI is inherently alien and there's no reason to expect it to think the way we do.
8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc)
So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
We can't know if we've started the process to summon alien entities from another dimension through a rift in spacetime, either.
On the other hand, we don't have any scientific reason to believe that superhuman AGI isn't possible. We know AI is possible, because we have it right now. What's preventing it from getting smarter than us? The human soul?
To clarify, most believe we likely have started on the path to superhuman AGI. We just don't know how or when we'll reach an arbitrary threshold where it becomes dangerous. There are many finish lines and we can't see any of them, but we're running as fast as we can down the path of least resistance and hoping we don't cross a finish line we don't like.
Sorry, what? According to which scientific theories is AGI possible, and what principles are those theories based on? What testable hypotheses say that it is, and which experimental evidence has been produced in favour of AGI being possible?
Human-level intelligence is possible. See: humans
Do you think it's physically impossible to recreate human-level general intelligence artificially?
Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Is there any physical law whatsoever that even hints that intelligence should be able to create AGI at its own level, but in no way more intelligent?
I never said there was a physical theory that predicts superhuman AGI. But in contrast, there are other concepts such as time travel which are prohibited by certain theories we have. It seems very unlikely for time travel to be possible. Superhuman AGI is not prohibited from existence.
I have no idea. I have no basis for concluding it's either possible or impossible.
> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
No, I think a godlike being could materialise an entity with superhuman intelligence, or natural selection could evolve one (perhaps even on earth -- perhaps it already has!).
> Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Even assuming I believe both the above are possible that doesn't mean I think it's possible they could occur on planet earth within any meaningful time frame. I also think that interstellar travel is compatible with the laws of physics but just because we've sent humans to the moon and robots to mars we have no idea whether interstellar travel is feasible or when it might occur. For me the response to the advent of GPT is like people saying "wow, we've sent men to walk on the moon -- if we keep going people will soon be visiting mars/alpha centauri!".
There's no rational basis (scientific, logical, anything) for knowing whether AGI is more akin to mars in this analogy (and we're really quite close) or to alpha centauri (and we're nowhere near).
> No, I think a godlike being...
Superhuman intelligence is a reality in many fields already: chess, Go, self-driving cars at certain conditions, calculations and many more. This is not necessary to have it for everything at all.
And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.
I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one.
> Actually I'd love to know what your estimate actually is for AI becoming an existential threat
My estimate is: never. At least not in the form of some superintelligent AGI.
But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.
True in a fair number of cases...but, based on their actions to date, I doubt that Google cares enough about appearances to bother.
According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.
Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen.
AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control.
The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different.
The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.
Maybe that is an argument you've seen, but the more convincing argument I know of is that we have no way of knowing we've gone too far until after the fact. Doesn't matter how many times it goes right first, we don't even know how to determine if and when it has gone wrong. A powerful enough AI could deceive us into thinking things are going "right".
You have an extremely high risk dying in a vehicle collision but people dismiss the risk because it isn't immediately apparent that 40,000 people die every year in the U.S. from that very cause. Certain deadly events become common enough that they get ignored due to desensitization, or become categorized as individual and anecdotal tragedies because they aren't mass casualty events like a plane crash or a hotel collapsing. If a few thousand people die every quarter from AI lockups or failures that nobody could've predicted or prevented because the AI is fully operationally autonomous, it will be treated the same as vehicle deaths. And that is the true fear.
Many of those people were not tricked, tech CEOs are involving themselves with this and picking up Yudkowsky's particular strain of AI safety intentionally.
Instead of focusing on the very real problems AI causes and exacerbates today, problems that might affect tech companies' bottom lines if addressed, CEOs can control the narrative and appear proactive by buying into this brand of science fiction.
Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.
The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews.
I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent.
I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.
So what if he likes wearing a hat?
Consider Shrek. The story is made to subvert the trope that the ugly person is the villain, as it was in Disney movies.
But the real point isn’t some Platonic view of Beauty. It is about expectations. Like how advertisers use people wearing lab coats to subliminally feed credibility into the words of actors.
You may wish to be free of such bias, and you may even work to raise your own awareness and strive to critically examine the message of people. And let us all work towards such a world. But we should not pretend such a world exists today, despite our desires. It is a world that may exist in the future once we mature as a species.
Until that day arrives, advertisers and political movements will use our biases against us.
My claim isn’t that yudkowski is ugly, it is that he represents a trope that is being used to bias his message.
He appears to make an effort to look good just like most everyone else. If he was trying to look a certain way he might have tattoos or have died his hair. He doesn't, that I know of.
You seem to be policing a small recent decision he made to wear a hat.
Just like I didn’t create the trope of a scientist wearing a white lab coat. But if someone is doing a 2 min video sketch of a scientist, it is a good visual aid to include a white lab coat to communicate to an audience the role of a character.
And in the same way, a top hat would communicate a certain kind of character in a movie.
In the same way, there is a communication of a particular kind of character with other hat choices.
1. https://amp.knowyourmeme.com/memes/tips-fedora
But why should he not be allowed to wear one? Again, you're policing. It's a discriminatory dick move.
He can wear anything he wants. So can you. So can anyone.
If he wants to, and for all I care, he can dress up like a clown. Or dress up like a jester. Or he could wear Gucci or Prada. Or wear steampunk leather outfits like he is in the world of mad max. Or he could go clean cut preppy. He can wear a fedora and a trilby on top of one another.
But, by personal choice and temperament, he wants to dress as he dresses. It just so happens that his personal preference aligns with a particular trope. A particularly uncool trope in popular culture.
I’m pointing out that alignment. I am not making him dress that way and I am not forcing culture to judge him.
It takes a lot less effort to generate your superficial criticism than it takes the GP to reply.
Fortunately, he did reply in a sibling post, but he didn't have to.
I'm not buying this alternative reality where I'm trolling by pushing back on mocking someone's choice of a hat that is one of a limited number of common hat styles.
It's sort of unfixable because the taste-makers at NYT and WaPo sorted themselves into the conformist camp so early in their lives that anyone outside that camp is trivially portrayable as "weird" to them.
And yet he doesn't think it is worth the effort to take off his trilby hat in order to come across as marginally more respectable to the average person? When the way he is choosing to present himself is undermining his own message then yes, it is worth bringing up.
We don't have an AI safety problem, we have an human safety problem. Violent extremists are unsafe, greedy corporations seeking to suppress the free exploration and exercise of mathematics are unsafe. Existent language model chatbots are primarily just a public relations risk, at least so long as the government doesn't step in and create a monopoly that gives the market leaders choice of training biases the force of law.
> Yud is a doomsday cult leader (https://archive.is/YTqlI)
Are you really using that link to support your argument in the earnest? You know this was posted on April Fool's day? The text itself mentions it near the end, and it's also explicitly tagged as such directly below the headline/byline. At least warn people who're going to skim this.
> who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse.
You mean that bit from his article in Time Magazine? Where he talked about airstrikes against GPU farms might be necessary in case some country/group is violating the international moratorium he's proposing? I don't recall him advocating mass murder there, or ever. Of course, since he is insisting on a strong international agreement, military intervention - yes, including nuclear weapons - is part of the picture, because that's literally how international agreements are ultimately enforced.
As for the rest - the guy has been saying the same thing for way over a decade now. The mainstream interest only really picked up in the last half a year. But somehow he's in on it.
indeed, and by many who saw it regarded as a kookery, not justifying of much concern. If he wanted to preach to his S&M dunegon about the big bad machines that will punish you for all eternity, well not my circus, not my monkeys.
Thus far his followers who've turned violent have only directed it at their extended group [1] for not doing enough to stop the end of the world (and that one murder and attempted murder of their landlord [2], but I'm willing to believe that was a chance event and not a trend).
I now consider it an actual concern now that the profile has been expanded so dramatically there is a much greater risk that someone will fall for ℚ-anon who isn't so distracted by their S&M dungeon that they actually find time in their busy schedule to 'save the world'-- just like Qanon was some internet joke until the wrong person fell for it and rolled up to Comet Ping Pong and opened fire. That person only thought they were saving children from being sex slaves, whats someone powered by the belief that they are saving the lives all current and future children going to be willing to do? Their prophet says even nukes would be justified.
[And not just their lives-- there are fates worse than death, some of their scripture involves fearmongering about malevolent AI's simulating trillions of copies of people and subjecting them to the worst torture imaginable, just because they didn't do everything in their power to bring the malevolent AI into existence. Lets not even get into the fact that their proposed solution--since malevolent AI is inevitable unless we abandon advanced technologies "just don't do it" is only a short term fix at best-- to create an AI god embodying the values of the faithful so that it can permanently suppress all competing machine intelligence and engineering human life to 'optimize' away suffering.]
Maybe you live far enough away or don't have anyone you love peripherally connected to AI enough that you're not concerned that they'll be murdered due to this insanity, but I am a little worried and right now the best way I can think to mitigate the risk is to bring people's attention to the fact that they're normalizing and legitimizing a mentally ill position that expressly justifies the use of violence (and not just in "ha ha only serious" April fools posts) against a fanciful apocalyptic threat from the fevered imagination of a cult.
> Are you really using that link to support your argument in the earnest?
Yes, the post was presented as a joke but it legitimately reflects his position, as also reiterated in interviews and articles.
Moreover, it's a "rational" response to those who actually believe the claimed threat, particularly from the utilitarian ethics perspective advocated in ℚ-anon circles: they hold that it's ethical to murder a million to save billions (or trillions 'discounted to present value', counting all the humans that will never exist due to extinction-- "What matters most about our actions is their very long term effects.").
> I don't recall him advocating mass murder there
Nuclear weapons have been used twice in combat in human history. In one instance the death toll was a hundred and twenty nine thousand people, in the other it was two hundred and twenty six thousand people. The nuclear weapons in the US arsenal today are many times more powerful than those early weapons.
It's not like data centers are not blast hardened targets, no one has described NAFTA, the Paris agreement, or the berne convention as being enforced by nuclear weapons. In a serious publication with a readership of millions Yud unambiguously advocated using the threat of mass murder as a deterrent against parties performing too many mathematical calculations.
How many of his new followers will agree with the widespread death of a nuclear weapon a...
I only meant to imply the latter. I suppose I inadvertently applied the former by mentioning him juxtaposed to Noam Chomsky and Michael Moore, although it is a stretch for me to even see those two as "Thought Leaders" in the present day. Chomsky is some weird kind of anarcho-syndicalist and Michael Moore seemed to fade away during the Obama years. Neither are really "leading" anything these days.
> A week or two ago ... Yudkowsky was apparently a Complete Nobody.
There is nothing contradicting being a punching bag and a complete nobody. It's like a D-list actor whose entire career is to get his ass kicked by the hero in action movies. Or the B-tier heel in professional wrestling. You might eventually recognize his face and he may even achieve some cult-status fame. Their whole purpose is to display the might of the eventual hero so we often allow them to embody some apparent strength so that a sense of risk for the hero is maintained.
Yudkowski makes for a juicy target. For example /r/sneerclub has been around since 2015! The fact that the mainstream media has now identified him as a magnet for derision isn't surprising. No one has to know who he is. They just take one look at him, hear a couple of sentences from his mouth and they immediately form a strong opinion on his character.
> The mainstream interest only really picked up in the last half a year. But somehow he's in on it.
What do you mean "he's in on it"? I'm suggesting he is an unwitting participant. I have no doubt he actually believes he is the main character. He doesn't lack the ego for that.
FWIW, even though I personally really dislike Yudkowski as a person - I believe he has some interesting points to make. And frankly, it bugs me that he is so often dismissed without deep engagement with his arguments. But he is so lacking in media training and social skills that he is easily misdirected to look like the exact foolish/cartoonish caricature the media wants. Even in friendly podcast situations he comes off as unhinged.
And I honestly believe that boosting the signal of characters like Yudkowski is part of the mainstream establishments playbook for dealing with dissent. In comparison, Sam Altman's "aw shucks" performance in front of the senate, with favorable sound bites where he boyishly claims to have no financial stake in OpenAI, was positively wholesome.
Have you even read the article?
1. Bad people control and use AI to dominate/destroy the world
2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.
If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just
1. We might create ML models smarter than ourselves soon
2. We don't really understand how ML models work or what they might do
3. That seems dangerous
There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.).
The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.
Your burden of proof is backwards.
It is on the AI-doomers to explain why sci-fi concepts like “AI improving itself into a super intelligence” or “AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead” have any relevance in the real world.
The entire AI-doomer worldview is built off of unproven assumptions about the nature of intelligence and what computers are capable of, largely because thought leaders in the movement are incapable of separating sci-fi from reality.
If you watched the conversation around Meta with Facebook and Instagram, and then later the conversation around TikTok, almost nobody cut to the heart of the issue, which is that "algorithms" are being used to make decisions about what to show people, and that the algorithms themselves have been changed subtly hundreds or thousands of times over the years to maximize engagement, until the type of engagement being maximized has made people basically crazy. The same engagement-maximizing work will proceed with AI, and it will allow the software engineers and managers responsible for developing the "algorithm" abdicate even more responsibility for genocides and for mass hysteria because they can pretend like they have no control over the algorithm.
The same irresponsible people will pocket a bunch of money for all this work and they will maximize engagement until it blows up our entire culture. And because nobody understands it except the techno-cultists, nobody will hold them accountable.
I find it hard to fault someone who just gives the crowd what it wants.
See https://youtu.be/ld-AKg9-xpM?t=30 for a counterpoint.
https://www.alignmentawards.com/shutdown
Hell, so much so, that you can get awards, probably even tenure, for writing a compelling argument or providing technical research for how we could guarantee that we'd be able to turn off a super intelligent AGI. Thus far: no solution.
We have however found that even smart people initially think "just turn it off" will work.
Neil deGrasse Tyson used to think you could just turn it off. He has since changed his mind. So does every public intellectual I can think of that engages with the arguments in good faith. Even the OP article concedes AGI research into the possibility that AGI could kill us all is important (just that other things are more important right now). There is no argument made that "we could just turn it off" in OP and therefore we shouldn't do the research at all.
No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.
This isn't an argument, it is an assumption.
AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true.
> As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.
The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.
It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.
The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception.
The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.
Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent.
> The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.
It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy:
Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable.
Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us.
> It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.
I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.
But we've also never tried to wipe out ants. I bet we could if we felt it was really important. I bet a superhuman AI could do even better.
I have seen people make what is effectively this argument.
> Once it exists, its goals are more important than our lives.
That's a sci-fi trope. There is no reason that it must be the case.
> As a comparison, do humans ever care about the existence of an ant colony when they want to build something?
That would be more relevant if the ants had created us to be the way we are.
Ok, well I think they're foolish.
> That's a sci-fi trope. There is no reason that it must be the case.
No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so.
> That would be more relevant if the ants had created us to be the way we are.
sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.
The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing.
If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: https://www.youtube.com/watch?v=ZeecOKBus3Q
Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments.
Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.
It's also demonstrable that we have other physical ideas such as FTL travel that are indicated to be impossible by current theories of physics. If we didn't have the math saying otherwise, it would be an open question of whether we can travel faster than light; but we have pretty solid math math saying we cannot.
And what we're talking about is logical in nature. Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes. Therefore, how is it reasonable to say that creating human-level artificial intelligence should be assumed impossible until proven? Just because we don't have the technology for it yet doesn't mean we should assume it's impossible. Logically, it follows from the first two assertions that it is physically possible to create artificial human-like intelligence.
Once again, and I don't know if I've said it in this thread directly but I've had to post it over and over again, nobody is claiming that the latest AI developments represent an existential threat. That's a complete mischaracterization of the debate and reeks of bad-faith argument but I will give you the benefit of the doubt and assume you've misunderstood.
In fact, that statement conflates two things into one nonsense argument. What's scary about current AI is how quickly it is moving, indicating a short timeline to (future) dangerous AGI. Also, it is possible that future AGI will be dangerous. These are two separate, rather simple assertions. If you disagree with either, that's fair, but you have to address them separately.
In logic, sure. But we're not living in a system of formal logic. We're living in a very messy world, full of physics, chemistry, and even (shudder) biology.
Here's the important question:
What would you consider to be sufficient proof that AGI is impossible?
Like, hypothetically. Doesn't even have to be based on any of the current facts on the ground in our universe. What facts or arguments could possibly convince you that this is not something that can ever happen?
If the answer is "nothing that I can think of", then you're asking other people to provide something you can't even define.
(If the answer is "nothing, definitely", then that means AGI is, for you, unfalsifiable, and essentially falls into the same category as religion.)
> Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes.
And here, you're falling victim to the ambiguity in human language (or, at least, in English).
"Intelligence" is not a clearly-defined word in this context, and while you seem to be presenting it as meaning the same thing in those two sentences, I would claim that it does not.
In the second sentence, it is clear that it is intended to mean "thinking intelligently, in a manner and to a degree similarly to humans".
In the first sentence, it cannot mean anything about "thinking in the same manner as humans," because you are talking about "artificial intelligences" that have already been created, and none of them think in anything like the same manner as humans. The difference between existing "artificial intelligences" and either humans or a hypothetical AGI is a difference in kind, not in degree, and you (and many others) gloss over that when you talk about "artificial intelligence" in one breath and "human-level intelligence" in the next.
Saying that all we need to do is keep going on the same track we're on with LLMs and similar "AI" programs, and we'll very soon (or ever!) reach AGI, is very like saying all we need to do to solve NP-hard problems in P-time is to throw more hardware at it. Sure, you'll get faster at doing the thing you're doing, but without some hitherto-unforeseen breakthrough (proving P=NP in the latter case; figuring out how to make AGI in the former), you'll never bridge a difference of kind by increasing the degree of effort.
This is trivially easy? People used to believe FTL was possible. Then our understanding of physics changed and we now understand it is an impossible limit to pass.
Would you say people who believed FTL was possible before physics research showed it to be impossible were believing something "religious" and "unfalsifiable" ? Please, they believed something totally within respectable epistemic parameters given what they knew at the time. In fact, the speed of light as an upper limit to speed seems very counterintuitive and "silly" at first glance. Why should a limit exist?
Sure, it might well be some fact about intelligence that we wont be able to get to AGI ever just by throwing more compute and layers at it for the next few decades or any time soon given current technology.
But nobody has come up with a slam-dunk, with empirical backing, that indeed there is some limit and we won't ever get AGI despite current trends. In fact, the opposite has happened and people like Geoffrey Hinton who used to believe AI risk is fanciful and AGI is a long time away has changed his mind, given current trends. We don't have research giving us that FTL limit, so why believe the limit exists? Why do you believe the limit exists? Or do you believe the probability is so low that we shouldn't worry? Ok, what probability do you place on AGI being created in the next 100 years and what would you have to see such that your probability crosses some threshold that it makes sense to worry about it?
(PS: If aliens suddenly arrive and they have completely alien psychologies such that when we discuss our relative intelligences it makes sense to talk about it in kind and not degree, I really don't think anyone is going to care about this distinction. What's important is how well can these aliens achieve their goals relative to us. And if they can achieve any goal of theirs that comes in conflict with our goals then we can reasonably say that are more intelligent than us)
https://www.futurehealth.live/blog/2022/4/18/ai-that-disrupt...
It's a very compelling combination of ego and convenience.
To me it doesn't feel technical at all--just superficial use of some domain verbiage with lots of degrees of freedom to duct tape it all together into a story. He very much reminds me of Eric Drexler and the nanotech doomerism of the 80s and 90s. Guy also had all the right verbiage and a small following of fairly educated people. But where is the grey goo?
If we need a counterargument to Yudkowsky do we also need one to Drexler?
It's called life.
Drexler may have been off with the approach to take, but he isn't wrong about the fundamentals.
So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.
In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate.
Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods).
The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.
The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong.
Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai.
I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?
Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems.
Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.
In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?
I’m struggling on this side of the equation and find the hype and noise depressing.
I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive.
Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence.
Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will?
Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).