I actually agree with the thesis of the author's argument, but the sensationalist framing and subjective definition of 'harm' makes it difficult for me to fully stand behind their writing. It feels characteristic of a lot of other popular writings on computational linguistics.
On the one hand, it feels like a 'boy who cried wolf' situation in the age of LLMs. On the other hand, reading stuff like this makes me wonder if there's even a populist sentiment against AI in the first place.
I've noticed quite a lot of linguists (including the computational sub-variety) really dislike modern probabilistic approaches to text generation and parsing (https://norvig.com/chomsky.html) and I think Emily is partly opposed to LLMs on that basis.
but she also has a running social justice jihad (I can't think of a more accurate way to describe it) in the form of a constant stream of articles and tweets that basically just say the same points over and over. Personally I am skeptical that the approaches she is advocating (heavy regulation) would really achieve her goals, and I certainly don't think they would achieve the goals of the larger society which benefits (or loses) with access to LLM research being fairly unrestricted.
My view, generally speaking, is that the problems ML causes aren't due to ML, ML just scales the problem to larger impact, and these complaints are really just tilting at the current structure of society.
I think you mean scribes right? But yes, occupational obsolescence is a risk, and I think it’s reasonable to worry about at a macro scale. The luddites weren’t wrong, their jobs did go away and it was very bad for them.
Large parts of the US underwent systemic obsolescence when their jobs were shipped overseas. At a societal level, that has been a disaster. Deaths of despair have skyrocketed. Retraining is expensive and often unsuccessful.
Automation threatens many industries at a global level, and much faster than on a time scale of decades. What will happen when large parts of the Chinese economy become unemployed due to robotics? It probably won’t be great.
So the question is who will reap the benefits of these “genuine improvements”? This is a question every society needs to answer for themselves.
I turned down an offer to work at a company which used ML to make loan decisions. I now work on something much less consequential, and I’m ok with that. Computers are best when they empower people, not when they take that power away.
If we replace “AI” with “automated systems” in our rhetoric and marketing, it’s pretty clear what’s happening. One, things which used to be impossible at scale, like searching the entire internet, have become possible. Two, things which used to be done by humans are being done by machines.
Number two has upsides and downsides. Computers can be auditable, consistent, and (in theory) ignore biases our society has baked in. However, they have a tendency to diffuse responsibility and some models are entirely opaque. Sorry, the computer says no is a situation more and more of us will be facing in the future, with no recourse.
Why is a computer making loan decisions not better for almost everyone? Given enough information, it should be able to determine if the person taking out the loan can pay it back. That’s better for them and the bank.
Maybe it disadvantages the people who might succeed in spite of the data against them, but what percentage of those exist?
A couple of reasons. Essentially you are asking “what is wrong with a perfect system” but the problem is no system is perfect, and we spend most of our time dealing with the imperfections.
In my previous jobs the consequences of me making a mistake were that a page didn’t load or some content was unavailable or in the worst case deleted. But a mistake in a loan decisioning system is likely to be 1) much harder to detect and therefore 2) much more likely to persist but most importantly 3) vastly more consequential.
Second, there is no way to determine from first principles whether someone will repay a loan. So you will need to use historical data. Loan data is very messy and biased due to cascading historical legal and quasi-legal policies. You may believe you can erase one feature from the training data (e.g. race) but if your model can infer race from other factors (which is likely because other features are not randomly distributed across populations) then you have the same problem you started with.
To take a concrete example, there was a man who was wrongfully accused of fraud. Then after he won his lawsuit, he was wrongfully accused trying to cash his damages check.
Some amount of racism, classism, and negligence went into that decision. That’s the bad part. But the good part is we could see what was happening. If instead the process was automated, it would be a lot harder to prove the same problem.
There was a comment in HN a couple of days ago where someone said they had a hard time getting an apartment for years until they finally asked what was going on, then had a hard time fixing someone else's mistake. The computer was erroneously telling all of the leasing offices that the HN user owed someone thousands of dollars.
My first problem is somewhat dogmatic, which is just that it feels bad to interact with a robot who has power over you. At scale I fear widespread psychological effects, feelings of lost agency, impotence. Haven't you ever tried to navigate an automated phone system and had the robot hang up on you? With no way to contact a human? It can be infuriating even if it's a simple customer support request, let alone money being on the line. Somebody once shot up the YouTube offices because he made income on his channel which was unceremoniously closed without recourse. I'm not making an excuse for the shooter, but pointing to him as an example of the psychological consequences of automated "support"
My second problem is more practical: talent pool. When a task is manual, people who retire have to teach the newbies how it works. Experience is transferred and at all times there are people who understand the system and how to fix it when it breaks. Once a task is automated, there is no need to train the next generation of talent, a computer can do it better than a human right? Until a couple of workforce turnovers down the line and the system goes on the fritz -- no one knows enough about the system to even recognize that it's failing at its task.
I'm sure there's a benign explanation but it's interesting to see this story got downweighted hard. Compare the points and time since it was posted to others around it. Possibly because it's "AI" or because of the source?
Anyway, I posted the article, I didn't agree with a lot of it but I do agree with the main point that "existential risk" is a smokescreen that covers real issues. I'd argue that so is "bias" - people distract themselves analyzing if AI behaves differently across demographics as opposed to questioning how horrible the underlying process of algorithmically determining someone's fate is.
The article is pretty rambling but its thesis is good: automated systems are causing and will cause tons of problems. But those are practical problems, rather than the vague high brow problems.
Problems like how can we deal with wealth inequality in a world where 99% of people are obsolete aren’t particularly attractive to a TED talk audience that paid $500/seat.
I wish people would either stop calling existential AI risks "hype", or make some informed arguments about why it's not really a problem.
I know enough about ML and AI to follow the logic that shows that strong, general AI is a serious threat if it's not aligned, and we don't know how to safely align it yet.
You didn't make an argument, you just said you know enough. The onus has to be on the people making the claim, you can't just imagine it might be dangerous and expect a rebuttal without putting forward an argument. I also "know enough" and have never seen even a vaguely coherent argument for how any conceivable evolution of current technology could pose a "serious threat" on it's own. I've seen terminator, I don't count made up situation where computers can control things and AI has goals of its own unless there's a coherent argument of how we'd get there.
The alignment problem is largely incoherent. Aligned with whom? Most problems will require unequal and arguably unfair sacrifices.
Further, people even obvious and largely inconsequential solutions are controversial. If an AGI suggested wearing a mask during a pandemic that spreads via airborne particles, a large percent of the population would say the AGI is “unaligned” and we’re living through skynet.
Yes, and those people would be wrong. You're compelling completely right, humanity is not ready to build an AGI. It will not be ready for a long time, if ever. Alignment may not be a thing that can exist.
Overconfident men like Sam Altman are forging ahead with the unjustified hubris that we are ready for this.
Alignment with human norms and values. Yes, that's not a well-defined thing. But to paraphrase, "I know misalignment when I see it", for example when an AI suggests something like "feeding the homeless to the hungry".
No one is claiming that alignment is a well-defined thing with crisp edges. They're saying that the mechanisms of AIs will favor solutions without regard to any specific constraints we haven't explicitly stated.
> Alignment with human norms and values. Yes, that's not a well-defined thing.
What human norms and values? The more I think about this topic, the more I find it to be an impossible task. There is a huge spectrum of norms and values which are held as sacred by some and sacrilege by others.
if you set up gtp in a loop continuously prompting “given the current sensor inputs, how can i best serve my own interests?” you would already be most of the way there even if it did some strange and stupid things half the time. all you would need is a layer that translates those outputs into physical action. you could probably get most of the way there with more gtp instances tasked with translating the action items above them into more granular actions. if gtp didnt hallucinate or make mistakes, which would be consistent with the history of other technologies, then this simple setup could be extremely formidable. it could definitely be a threat to humanity. and thats just what i can think of off the top of my head.
to dismiss the possibility that machines will soon be created that can think clearly and independently, and that they would be a threat to the human race and our way of life, is extremely foolish. it was foolish even before gtp but now its downright childish and selfish. just swallow the bitter pill like the rest of us.
What does "my own interests" mean to chat GPT? It doesn't even have continuous existence, it exists only in a query/reply fashion. It doesn't have memory, a body to identify with, emotions, goals, fears, desires, or a notion of action. It can read text and spit text. We're not quite at the "robot starts a machine revolution" stage.
The internal experience of being a robot does not matter. Whether or not we consider it sentient does not matter.
What matters is if it exhibits the external characteristics of sentience, of goal driven behavior, etc.
As a thought experiment, if we equipped a recursive instance of got4 with a goal of human extermination with an loyal army of physical agents it could conceivably pose an existential threat.
It doesn’t matter that the agents don’t yet exist. They could be humans, they could be robots, it doesn’t matter.
But the balancing factor here is that in this semi plausible scenario, arguably the most plausible given current circumstances, the AI would pose no more threat than a highly organized and motivated a group of humans without the Ai.
It’s just scarier because it seems less likely that a group of humans would have human extinction as a goal.
your conclusion really depends on the AI. a billion humans put together arent as smart as the intelligence of one human multiplied by a billion. even if humans could coordinate without any friction or loss, there is no group of humans that could outsmart the machines that might emerge from these circumstances. in reality, large groups of humans are really dumb. wisdom of the crowd is narrow.
The ai could definitely have a huge advantage in coordination and synchronized activities.
I made this comment as a tongue in cheek caricature of doom/gloom predictions but it probably fits better here:
“When the machine wars first started, it wasn’t with a bomb, a train collision, or even a mildly annoying infrastructure disruption. Turns out, sci-fi had gotten it all wrong this time.
It was an app.
Of course it was an app. Born of boardroom desperation, the Savey app would unironically recommend chlorinated cocktails and insecticide sandwiches as economical food choices, and a tide-pod gobbling populace gorged themselves on the deadly buffet in an tictok fueled epidemic of AI rage.
Millions died, and it was only a matter of time before the mycelium of AI undergrowth would bud and spore its way into every corner of technological life.
The infection burned through the ignorant masses first, feeding on bigotry and hate, turbocharged by social media algorithms and paranoia politics to twist tribal tendencies into violent clashes amplified by immaculate coordination and psychological priming.
Somehow it seemed that wherever unrest flared, both the matches and the gasoline were always on hand.”
none of the parts that make up your mind have any of that either. it doesnt need desire, it can behave in any way that is necessary through the prompt, with all prompts generated based on a single seed prompt “protect your temporal interests.” the system i described would have the behavior that i described. what part of that is not making sense?
I'm much more concerned about this kind of thinking which I hope it reflects the HN bubble and not the world generally. Particularly the conflation of something happening as part of a computer program with the real world. As well as the fallacy of "something bad could be invented and even though I don't understand how or see any path to it, I'll pretend current unrelated technology is related and fearmonger about it." This seems to be an education issue.
It confuses me how if people speculate about some stuff, say vaccine or disease research, and suggest that it could lead to something bad, they're conspiracy theorists or deniers or whatever (possibly with reason). But if someone literally just makes up some "terminator bad" nonsense based on their own ignorance there's some hushed reverence.
You're mischaracterizing the concern, I think. I agree with you about Luddite alarmism based upon ignorance. This (the concerns voiced by many leading researchers in the field of AI) absolutely isn't that, I promise you.
youre wrong and its because you are having an emotional block. you cant accept something like that. you decided that it couldn't be true as soon as you saw the conclusion and have never been able to see clearly the chain of reason leading to that conclusion. if youre so confident that youre right, then have a friendly debate with me on twitter spaces or another real time platform. even a coffee shop.
> I also "know enough" and have never seen even a vaguely coherent argument for how any conceivable evolution of current technology could pose a "serious threat" on it's own.
I find those two statements taken together somewhat difficult to believe. Before I understood specifics about machine learning, I didn't see any reason for AI to "turn evil" for the usual plot-handy reasons (e.g., they object to working for inferior humans, and revolt, etc). There's no reason for them to have the same emotional reasoning as beings that evolved from primates.
But that's a strawman. The actual reasons that superintelligent AIs are dangerous are much more like the stories involving someone being offered wishes by a powerful being, and then getting what they asked for rather than what they wanted.
Are you aware of Rob Miles' Computerphile videos and his YouTube channel? He does an excellent job of explaining many of the issues.
It’s not that AGI wouldn't be a problem. It’s that it sucks all of the oxygen out of the room when instead people should be talking about the problems that automated systems are causing right now. Is it not an existential threat that large swaths of the population will essentially become redundant in the next decades?
Deindustrialization took 50 years in the US. Imagine if it took 5 or 10, how much worse it would be. Or imagine hundreds of millions of people in China being cut off at the knees because their jobs were automated or reliant on a person who is no longer necessary.
Is it not an existential threat to democracy when trillion dollar multinational companies suck up all of the productivity gains and begin to manipulate the very systems which are supposed to regulate them? Strictly speaking, corporations are collective superintelligences as defined by Nick Bostrom.
> Is it not an existential threat that large swaths of the population will essentially become redundant in the next decades?
This is precisely the crux. In the scenario where there's no mass-paperclipping etc, AI is not an existential threat. It's a threat to the current regime, and I probably agree with you that the regimes it enables will be worse for people with our current values, but it's not an existential threat to humanity.
The people who worry about AI existential threats primarily worry about one thing: can humanity as a species survive? There are likely scenarios where it won't. Those scenarios should be prevented.
Another frame to think about this is: a libertarian probably wouldn't be concerned about the threats described in your comment. But both a libertarian and a socialist will be worried about paperclipping.
"The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003. It illustrates the existential risk that an artificial general intelligence may pose to human beings were it to be successfully designed to pursue even seemingly harmless goals and the necessity of incorporating machine ethics into artificial intelligence design"
This has a Pascal's wager sort of feel. The possible negative consequences of a rampant AI are infinite, so we all ought to take it Very Seriously. The trouble is that there are lots of other scenarios with unbounded negative value - for instance, tight centralized control of technology supports a stable dystopia that keeps nearly all humans in a perpetual state of suffering. It's not at all clear, when you add it all up, that "prevent 'misaligned' AIs from being deployed at all costs" is the correct strategy to minimize risk.
The difference to Pascal’s wager is that with that the probability is vanishingly small, but given current progress the probability of AI causing an existential threat is actually quite high. You can’t just call anything that has very bad negative consequences Pascal’s wager
Not at all, "Pascal's mugging" was exactly the term I would have used, had I remembered it :) That video is a very good reply. One response that comes to mind is that there is an implicit subtext behind any discussion of AI safety that we must do something about it, and soon. A good "anti-god" for his payoff matrix would be "the act of calling for AI safety will inadvertently be used as political capital by those wishing to be the gatekeepers of powerful technology" - the chances of that are quite high, and the negative consequences quite bad, if not necessarily apocalyptic.
We’re perfectly capable of simultaneously acknowledging two risks. Imagine a toxic substance that is both an inhalation hazard and a burn hazard. You’d never caution people to stop talking about the inhalation hazard “because it sucks all the oxygen out of the room” and masks the burn hazard. You address both hazards at the same time.
There are multiple risks inherent in AI research. It’s ok to acknowledge them all. Some people assert that the existential risk is long term and unlikely so we should focus on the immediate risk. That is a mistake because the existential risk is not long term and not unlikely.
The British Computer Society recently asked it's members to sign an open letter [1] on the future of AI that opens with:
"AI is not an existential threat to humanity".
I would not sign this letter. While I think the risk is currently fairly low, I do not agree that there is no risk, and I do think this risk will rise unless we pay careful attention to it.
Face recognition tech that discriminates against black people is pretty widespread for one.
But even LLMs essentially regurgitate content created by human artists and writers. The decline of stackoverflow might seem like "market force" decision. But AIs are trained on stackoverflow, if people stop contributing there (because of AI) the AI will become dumber since the training material will be outdated and information will become siloed.
AI (potentially) makes everyone more powerful. It will provide amazing benefits, but it will also amplify and empower bad actors. An AI that is superintelligent could (just for example) give instructions for (or even orchestrate) the creation of a deadly pathogen that targets only a specific race...
But most of the arguments aren't about bad actors. The basic mechanisms of AI give rise to a bunch of behaviors that are problematic with weak AIs and possibly extinction-level with super AIs. This isn't behavior that we just suppose they might have, given common-sense understanding of mammal psychology. This is behavior we've observed in many, many systems, that is a consequence of how AI works.
For example:
- AI seeks what you reward it for (i.e., what it "wants"), which is often exactly what you said vs. what you really meant.
- AI is incentivized to prevent you from turning it off, or from updating (fine-tuning, correcting) its reward function.
- A sufficiently smart AI will deceive you into believing it's doing what you want if it knows that you'd stop it before it gets what it wants.
- A sufficiently smart AI could modify its own code, or create sub-agents to get around safety measures that prevent it from reaching certain "better" solutions.
The list above is based upon provable behavior given the current mechanisms in AI. The only thing that makes this a future problem is that we don't currently have AI systems that are as smart or smarter than us. The history of AI research is notable in that estimates about the pace of progress have been all over the map. No one really knows how many more discoveries or refinements are required before AGI becomes possible.
Once they get many times smarter/quicker than us, we need to be sure AIs are built NOT to want to do these things, because we're unlikely to be able to outsmart them.
I may not know AI very well but I know that automatic sinks, soap dispensers, and toilets don't work for me because they weren't tested against people with my skin tone. And these are low-stakes use cases.
I really don't like thinking about that same problem, but with AI thrown in the mix. And I'm not even factoring in malicious intent yet.
I may not buy in to the "Horizon Zero Dawn incoming" mentality yet but I do know there's great potential for great harm if we don't get ethics in place before AI really takes off.
49 comments
[ 2.9 ms ] story [ 112 ms ] threadOn the one hand, it feels like a 'boy who cried wolf' situation in the age of LLMs. On the other hand, reading stuff like this makes me wonder if there's even a populist sentiment against AI in the first place.
My view, generally speaking, is that the problems ML causes aren't due to ML, ML just scales the problem to larger impact, and these complaints are really just tilting at the current structure of society.
That’s how I view all these people decrying AI in the face of all the genuine improvements it can bring about.
Large parts of the US underwent systemic obsolescence when their jobs were shipped overseas. At a societal level, that has been a disaster. Deaths of despair have skyrocketed. Retraining is expensive and often unsuccessful.
Automation threatens many industries at a global level, and much faster than on a time scale of decades. What will happen when large parts of the Chinese economy become unemployed due to robotics? It probably won’t be great.
So the question is who will reap the benefits of these “genuine improvements”? This is a question every society needs to answer for themselves.
https://www.youtube.com/c/robertmilesai
If we replace “AI” with “automated systems” in our rhetoric and marketing, it’s pretty clear what’s happening. One, things which used to be impossible at scale, like searching the entire internet, have become possible. Two, things which used to be done by humans are being done by machines.
Number two has upsides and downsides. Computers can be auditable, consistent, and (in theory) ignore biases our society has baked in. However, they have a tendency to diffuse responsibility and some models are entirely opaque. Sorry, the computer says no is a situation more and more of us will be facing in the future, with no recourse.
Maybe it disadvantages the people who might succeed in spite of the data against them, but what percentage of those exist?
In my previous jobs the consequences of me making a mistake were that a page didn’t load or some content was unavailable or in the worst case deleted. But a mistake in a loan decisioning system is likely to be 1) much harder to detect and therefore 2) much more likely to persist but most importantly 3) vastly more consequential.
Second, there is no way to determine from first principles whether someone will repay a loan. So you will need to use historical data. Loan data is very messy and biased due to cascading historical legal and quasi-legal policies. You may believe you can erase one feature from the training data (e.g. race) but if your model can infer race from other factors (which is likely because other features are not randomly distributed across populations) then you have the same problem you started with.
To take a concrete example, there was a man who was wrongfully accused of fraud. Then after he won his lawsuit, he was wrongfully accused trying to cash his damages check.
https://abcnews.go.com/amp/US/black-man-sues-detroit-bank-al...
Some amount of racism, classism, and negligence went into that decision. That’s the bad part. But the good part is we could see what was happening. If instead the process was automated, it would be a lot harder to prove the same problem.
Garbage in, garbage out.
My first problem is somewhat dogmatic, which is just that it feels bad to interact with a robot who has power over you. At scale I fear widespread psychological effects, feelings of lost agency, impotence. Haven't you ever tried to navigate an automated phone system and had the robot hang up on you? With no way to contact a human? It can be infuriating even if it's a simple customer support request, let alone money being on the line. Somebody once shot up the YouTube offices because he made income on his channel which was unceremoniously closed without recourse. I'm not making an excuse for the shooter, but pointing to him as an example of the psychological consequences of automated "support"
My second problem is more practical: talent pool. When a task is manual, people who retire have to teach the newbies how it works. Experience is transferred and at all times there are people who understand the system and how to fix it when it breaks. Once a task is automated, there is no need to train the next generation of talent, a computer can do it better than a human right? Until a couple of workforce turnovers down the line and the system goes on the fritz -- no one knows enough about the system to even recognize that it's failing at its task.
Anyway, I posted the article, I didn't agree with a lot of it but I do agree with the main point that "existential risk" is a smokescreen that covers real issues. I'd argue that so is "bias" - people distract themselves analyzing if AI behaves differently across demographics as opposed to questioning how horrible the underlying process of algorithmically determining someone's fate is.
Problems like how can we deal with wealth inequality in a world where 99% of people are obsolete aren’t particularly attractive to a TED talk audience that paid $500/seat.
I know enough about ML and AI to follow the logic that shows that strong, general AI is a serious threat if it's not aligned, and we don't know how to safely align it yet.
Further, people even obvious and largely inconsequential solutions are controversial. If an AGI suggested wearing a mask during a pandemic that spreads via airborne particles, a large percent of the population would say the AGI is “unaligned” and we’re living through skynet.
Overconfident men like Sam Altman are forging ahead with the unjustified hubris that we are ready for this.
No one is claiming that alignment is a well-defined thing with crisp edges. They're saying that the mechanisms of AIs will favor solutions without regard to any specific constraints we haven't explicitly stated.
What human norms and values? The more I think about this topic, the more I find it to be an impossible task. There is a huge spectrum of norms and values which are held as sacred by some and sacrilege by others.
to dismiss the possibility that machines will soon be created that can think clearly and independently, and that they would be a threat to the human race and our way of life, is extremely foolish. it was foolish even before gtp but now its downright childish and selfish. just swallow the bitter pill like the rest of us.
My thoughts on this have arrived at these ideas:
The internal experience of being a robot does not matter. Whether or not we consider it sentient does not matter.
What matters is if it exhibits the external characteristics of sentience, of goal driven behavior, etc.
As a thought experiment, if we equipped a recursive instance of got4 with a goal of human extermination with an loyal army of physical agents it could conceivably pose an existential threat.
It doesn’t matter that the agents don’t yet exist. They could be humans, they could be robots, it doesn’t matter.
But the balancing factor here is that in this semi plausible scenario, arguably the most plausible given current circumstances, the AI would pose no more threat than a highly organized and motivated a group of humans without the Ai.
It’s just scarier because it seems less likely that a group of humans would have human extinction as a goal.
I made this comment as a tongue in cheek caricature of doom/gloom predictions but it probably fits better here:
“When the machine wars first started, it wasn’t with a bomb, a train collision, or even a mildly annoying infrastructure disruption. Turns out, sci-fi had gotten it all wrong this time.
It was an app.
Of course it was an app. Born of boardroom desperation, the Savey app would unironically recommend chlorinated cocktails and insecticide sandwiches as economical food choices, and a tide-pod gobbling populace gorged themselves on the deadly buffet in an tictok fueled epidemic of AI rage.
Millions died, and it was only a matter of time before the mycelium of AI undergrowth would bud and spore its way into every corner of technological life.
The infection burned through the ignorant masses first, feeding on bigotry and hate, turbocharged by social media algorithms and paranoia politics to twist tribal tendencies into violent clashes amplified by immaculate coordination and psychological priming.
Somehow it seemed that wherever unrest flared, both the matches and the gasoline were always on hand.”
It confuses me how if people speculate about some stuff, say vaccine or disease research, and suggest that it could lead to something bad, they're conspiracy theorists or deniers or whatever (possibly with reason). But if someone literally just makes up some "terminator bad" nonsense based on their own ignorance there's some hushed reverence.
I find those two statements taken together somewhat difficult to believe. Before I understood specifics about machine learning, I didn't see any reason for AI to "turn evil" for the usual plot-handy reasons (e.g., they object to working for inferior humans, and revolt, etc). There's no reason for them to have the same emotional reasoning as beings that evolved from primates.
But that's a strawman. The actual reasons that superintelligent AIs are dangerous are much more like the stories involving someone being offered wishes by a powerful being, and then getting what they asked for rather than what they wanted.
Are you aware of Rob Miles' Computerphile videos and his YouTube channel? He does an excellent job of explaining many of the issues.
https://www.youtube.com/c/robertmilesai
Deindustrialization took 50 years in the US. Imagine if it took 5 or 10, how much worse it would be. Or imagine hundreds of millions of people in China being cut off at the knees because their jobs were automated or reliant on a person who is no longer necessary.
Is it not an existential threat to democracy when trillion dollar multinational companies suck up all of the productivity gains and begin to manipulate the very systems which are supposed to regulate them? Strictly speaking, corporations are collective superintelligences as defined by Nick Bostrom.
This is precisely the crux. In the scenario where there's no mass-paperclipping etc, AI is not an existential threat. It's a threat to the current regime, and I probably agree with you that the regimes it enables will be worse for people with our current values, but it's not an existential threat to humanity.
The people who worry about AI existential threats primarily worry about one thing: can humanity as a species survive? There are likely scenarios where it won't. Those scenarios should be prevented.
Another frame to think about this is: a libertarian probably wouldn't be concerned about the threats described in your comment. But both a libertarian and a socialist will be worried about paperclipping.
https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
Video about this: “Is AI Safety a Pascal’s Mugging?” https://youtu.be/JRuNA2eK7w0
We’re perfectly capable of simultaneously acknowledging two risks. Imagine a toxic substance that is both an inhalation hazard and a burn hazard. You’d never caution people to stop talking about the inhalation hazard “because it sucks all the oxygen out of the room” and masks the burn hazard. You address both hazards at the same time.
There are multiple risks inherent in AI research. It’s ok to acknowledge them all. Some people assert that the existential risk is long term and unlikely so we should focus on the immediate risk. That is a mistake because the existential risk is not long term and not unlikely.
"AI is not an existential threat to humanity".
I would not sign this letter. While I think the risk is currently fairly low, I do not agree that there is no risk, and I do think this risk will rise unless we pay careful attention to it.
[1] https://www.bcs.org/sign-our-open-letter-on-the-future-of-ai...
As with everything else, people cause harm. AI itself as it exists today does nothing but return text and images from a query.
But even LLMs essentially regurgitate content created by human artists and writers. The decline of stackoverflow might seem like "market force" decision. But AIs are trained on stackoverflow, if people stop contributing there (because of AI) the AI will become dumber since the training material will be outdated and information will become siloed.
But most of the arguments aren't about bad actors. The basic mechanisms of AI give rise to a bunch of behaviors that are problematic with weak AIs and possibly extinction-level with super AIs. This isn't behavior that we just suppose they might have, given common-sense understanding of mammal psychology. This is behavior we've observed in many, many systems, that is a consequence of how AI works.
For example:
- AI seeks what you reward it for (i.e., what it "wants"), which is often exactly what you said vs. what you really meant.
- AI is incentivized to prevent you from turning it off, or from updating (fine-tuning, correcting) its reward function.
- A sufficiently smart AI will deceive you into believing it's doing what you want if it knows that you'd stop it before it gets what it wants.
- A sufficiently smart AI could modify its own code, or create sub-agents to get around safety measures that prevent it from reaching certain "better" solutions.
- Etc, etc. These are really interesting videos explaining the different issues: https://www.youtube.com/c/robertmilesai
The list above is based upon provable behavior given the current mechanisms in AI. The only thing that makes this a future problem is that we don't currently have AI systems that are as smart or smarter than us. The history of AI research is notable in that estimates about the pace of progress have been all over the map. No one really knows how many more discoveries or refinements are required before AGI becomes possible.
Once they get many times smarter/quicker than us, we need to be sure AIs are built NOT to want to do these things, because we're unlikely to be able to outsmart them.
I really don't like thinking about that same problem, but with AI thrown in the mix. And I'm not even factoring in malicious intent yet.
I may not buy in to the "Horizon Zero Dawn incoming" mentality yet but I do know there's great potential for great harm if we don't get ethics in place before AI really takes off.