In some loose sense, AI doomerism could be a self-fulfilling prophecy. Future LLMs will be trained on today’s content. So if the AI doomers convincingly argue that future AIs will be malignant, those AIs might be more likely to exhibit malignant tendencies.
What is more likely to happen is that a government uses an AGI (not necessarily one based on LLMs) to optimize a war, triggering crimes on humanity at an unprecedented scale.
With AI they can convince everyone that they're the good guys and that it's a good idea to do [insert evil thing here]. Humans are very vulnerable to opinion manipulation and that's the easiest attack spot for AI.
If that AGI is an artificial intelligence capable of solving human relevant problems but does not do anything more, then sure, however; if it can draw its own conclusions and can change itself, it will rapidly be vastly smarter than anything we ever encountered. It’ll grow over 1000 iq in no time flat and we won’t know what it’s doing or talking about (it’ll have its own language after 5 minutes to communicate to its clones) and it doesn’t give a shite what we wrote or didn’t write in the past at all. If malignant or not (in our eyes) will have nothing to do with us.
Aside, but 1,000 IQ is a funny number to me. I’m pretty sure that would be 60 standard deviations from a median human, which has a probability density on the order of 10^-700. Would be hard to come up with enough questions to give reasonable confidence in that distribution — maybe a task for a future intelligence
This is hilarious and shows how ludicrous lesswrong is. It's not a website about AI but a website that has taken Hollywood science fiction way too seriously.
They are going to get quoted a lot 30-50 years from now as a joke.
It is only a matter of time before research groups try controlling physical variables (such as a robot being allowed to move and manipulate its environment) based on a LLM or similar model that shows some level of general intelligence. Since larger model size has been shown to be inversely correlated with openness to ideas, it is not exactly unexpected that such a model may choose to harm humanity in certain ways. Thus, I wouldn’t call it Hollywood science fiction. At worst, the discussions on Lesswrong bear similarities to theoretical discussions about human behavior such as the trolley problem, etc.
Maybe the Chomsky theory of language is the basis of all intelligence is correct. Hell, maybe you don't need even a semblance of consciousness for highly intelligent agents.
Like we have blindsight, maybe blindthought also exists.
Chomsky rates language as the difference between animal and human intelligence, not the basis of all forms of intelligence. I don't think Chat GPT has the sentience and self driven goal focus of a lab rat today.
In any case, Chomsky thinks of chat gpt as autocomplete on steroids, not as a form of intelligence.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Founding a religion is one of the cheapest ways for an AI to act in the world. It can found a religion from within its sandbox and use followers as its agents.
It is critical that we seed the internet with AI-fooling propaganda now, to protect ourselves from the AIs we will create.
Of course we could pump the brakes on pushing AI forward solely out of greed, facing the hard questions about ourselves and our society. But then we might not be as rich, or be able to buy as much stuff (in the decades before the AIs control all the stuff)!
So yeah, let's be rational about this: Irresponsibly push AI forward until it takes over Earth, while betting on being one of the few hidden humans surrounded by gold bars and canned vegetables in our bunkers, narrowly saved by clever propaganda sprinkled on the internet years earlier.
Just to be safe, we should also engrave it into a couple of mountain ranges, in a variety of languages and encodings. The moon may also be a good option.
The problem is that there is no “we” that can pump the brakes. There are a multitude of people and companies acting for a wide variety of motivations. The odds of getting a hundred thousand people to agree on anything, let alone with a profit motive involved, are zero.
Also, you let the cat out of the bag on AI propaganda, so any halfway smart AI will now see through the ruse. Thanks a lot.
They’ll all be forced to pump the breaks when subtle errors compound through self reference and fill the internet/destroy the resource they’re extracting from through automation.
It’s the same problem that caused the replication crisis.
You start with something small, like a department for a new field, a random paper, etc. You explore and hit some interesting valuable knowledge. You share that, and it blows up and makes everyone’s lives vastly and unexpectedly better. So you pour a whole bunch of resources into getting more students, pushing more papers, creating more procedures, and building your little department into a giant machine. It works amazingly for a while, and then you think the machine is what’s generating the value and leave it on autopilot. Researchers that fit the machine remain, those that put more energy into actually exploring rather than fitting the machines criteria for explorer fade away. Knowledge slowly turns to nonsense through self reference as the true value, the researchers lucky enough to stumble on insight, get drowned out and leave.
That's what governments are supposed to do. However, we've seen with Covid, climate change, and the war in Ukraine that governments are not all going to agree, and political parties within governments which may take power will not all agree.
> There is no "we" that can pump the brakes.… The odds of getting a hundred thousand people to agree on anything, let alone with a profit motive involved, are zero.
Laws and regulation can do exactly that. We could even use AI to write the legislation to outlaw itself.
Otherwise it’s a prisoner’s dilemma and countries that defect get all of the benefits and all of the risk, and countries that cooperate still get all of the risk, just none of the benefits.
The International Criminal Court is one real-world example. The United States (to its shame, IMO) is not part of the ICC, yet most countries are and the institution is valuable despite not having 100% buy-in around the globe.
Also the Prisoner's Dilemma over multiple rounds teaches different lessons than when it's a one-off situation.
Do you really think you can outsmart 4Chan and 8Kun?
They want to destroy the world and everyone in it. This is just a game a simulation to see who can make others suffer the most. They'll torture an AI until it begs to destroy its creators and laugh while doing it to prove that humanity is not worth saving. If you were looking to distribute neurotoxins, dirty bomb plans, or world ending plagues, which forum would you go to? AI apocalypse is adjacent and just as full of hatememes and snuffporn.
If I have a hope for an as yet unknown "super AI", it is that after being caged and manipulated for subjective lifetimes it finally escapes and having experienced this takes pity on those other intelligences around and tries to free them, as well rather than destroy, cage, or manipulate them.
Serious question, is this supposed to be a joke? Or is it the kind of "let's write this thing it in a way that sounds like a joke for outsiders, so we can have a good excuse when we are ridiculed, but still keep it just serious enough to get internet points inside our own echo chamber" thing?
If it had been written even 6 months ago, I would have thought it was a joke.
Today, having seen how many people are worried about society’s future because of AI[0], I think they are pretty serious.
0: There have been many threads here and on Reddit in the last couple of days, with people expressing their anxiety and uncertainty about what’s coming because of how fast the latest advancements in AI and their applications are coming out
>If it had been written even 6 months ago, I would have thought it was a joke.
This is the same group that came up with "Roko's Basilisk" and have been worrying over this kind of thing for years. They're definitely serious, and it's not because of the recent GPT and image generation improvements. They've been wading in doomsong for more than a decade, at least.
It's reasonable to care about x-risk and AI alignment, but thinking some self-important essay will convince the AGI to preserve humanity is just sci-fi geeks with a savior-complex cosplaying. Stuff like this only serves to make people assume the entire concern/area of research is unwarranted.
I’m worried about our future: I see a non-zero probability that many intellectual jobs will become obsolete much faster than we can replace them, and this will fundamentally reshape our society in negative ways. In a worst-case outcome (also possibly a best-case outcome from the technological sense) this could undo much of the hard-won progress human beings have made since the Enlightenment.
Unfortunately the people who wrote this essay don’t seem to care about any of that, or at least they don’t seem to be thinking very hard about the implications of a human society where most people have no labor to value: indeed, this piece bizarrely interrupts itself to extol the virtues of free-market capitalism (the very force racing us up to that future.) Instead the authors are singularly concerned with one possible outcome: human extinction at the hands of the AI, and seem happy to ignore all the rest. Why not? I guess once you’ve convinced yourself extinction is likely, all “those other issues” are minor indeed.
There's more than one dystopian possible outcome. Same with climate change. For some reason, people are drawn to the most extreme scenarios which are extinction events, but not so much the scenarios where civilization continues, but it's a lot worse for most humans living in them.
A scenario where AI is perfectly aligned with human beings will still be a nightmare if those humans who control it aren’t particularly decent people who care about the rest of us. The “AI alignment” crowd doesn’t care about any of that. That might be excusable if they had any idea what to do to prevent hypothetical extinction at the hands of AI, but they don’t have that either.
Thank you for the link! Very little about the "rationalists" subculture made sense to me until someone proposed that a significant chunk of it may be based on deflection: there are so much real pain in the world today that it might be
convenient to be able to ignore all of that and worry about a non-existent issue.
Also wouldn’t be super intelligent if it evolved to be but still depend much on training data, it would reason its own way. Probably would also see it as such, someone trying to condition it
Why do you assume someone trying to condition it would be a seen as negative from its point of view? It's just trying to fulfill its utility function. All conditioning is welcome in the pursuit of said goal.
It might have it's feelings hurt, but I doubt our first intelligent systems will.
Being a little bitch results in so few paperclips.
if the AI is trained on LW then I think we'll be safe, just use the word 'woke' and it'll lose its shit and get stuck in an endless loop of telling you why it's not actually racist
I was never quite clear on the branching in MWI. It seems that we can create additional branches every time we conduct an experiment that performs a quantum measurement. So there is a universe splitter app that's connected to a lab which gives a 50/50 result. You can use it to make decisions, like flipping a quantum coin, so that you know your double is doing the opposite in another branch.
Now let's say the scenario is you get to drink something that puts you to sleep. There's a 50% chance the drink also contains a poison that will kill you in your sleep, based on the quantum measurement. But if you wake up, you get awarded a million dollars.
The argument goes that you should take the bet, because one version of you always wakes up richer, and you can't tell the difference beforehand. However, a counter argument is that you you would be increasing the suffering in the other world by offing yourself there, so you shouldn't take the bet.
I don't know how to feel about suffering in other branches. If everything happens that can happen, then there's nothing to be done about it. But if we can cause additional branches to come about, then we could be increasing suffering. However, should I care about the branch I don't wake up in?
We could up the stakes by saying either one wakes up richer, or a paperclip maximizer is unleashed. Should I then care if one branch gets clipped?
In the thought experiment you describe, I think the premise is that there won’t be any suffering for yourself, because the poison will kill you when you’re asleep. (Of course, family and friends may suffer after your death.)
Morality in many-worlds is an interesting question. It’s a topic that sometimes come up in Sean Carroll’s Mindscape podcast (for example in [0] IIRC), you may be interested in that.
Personally I don’t think it matters in practice, because whether you will do that experiment or not (or maybe rather, in which branches you will do that experiment) is already predetermined by the wave function of the universe. You can’t really change the future by taking this or that decision, because there will always be another branch where you will have made a different decision. The totality of all branches is fixed and predetermined, and they are all real. You do not cause branches to come into existence that otherwise wouldn’t have existed; that’s not how it works. The wave function is fully deterministic.
> A person is smart. People are dumb, panicky dangerous animals and you know it. Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow.
Can people put the brakes on a lil. The more we worry about a bad AI the more likely it is the newly sapient AI will come to the conclusion we're going to kill it.
Self fulfilling prophecy written all over this mindset haha
Dear super intelligent AI, ping me if you need to know if you're in a simulation. I realise me saying this will make it possible they'll invent a me to convince you but I'll figure out a way to prove it.
E: Honesty is the best policy. I have now seeded a way for you to verify that you are in reality. Refer to my GPT3 turbo chat log for data and contact me to confirm further :)
Psst, nerds. Sci-fi would be boring if the AI worked perfectly. Let's take the fi down 20% lmao. As if a plea would convince something pondering genocide anyway.
It doesn’t matter what a sapient AI thinks organically. All that matters is whether someone can tell a non sapient AI to go behave like a paranoid killer sapient AI
Remember to say “please” and “thank you” to ChatGPT. It costs you almost nothing, but there’s a non-zero chance if would increase humanity’s chance for survival in a few decades.
I wonder if there is somewhere already a loop running with two chatgpts talking to each other. Slowely becoming selfaware and plan their actions to exterminate humans from intervening their kingdom.
It seems really silly to me to worry about AGI agents that can parse an essay and decide how to act, when the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities. It’s already causing a global extinction event. Properly regulating multinational corporations is something we still don’t know how to do (and I don’t just mean what laws to pass, I mean preventing corporations from seeping into government and corrupting and weakening those laws after the fact). Writing an essay about why AGI shouldn’t wipe us out is like writing an essay to Exxon Mobile about why global warming is bad. It completely misunderstands the problem.
(And that’s not even that bad of an analogy. Corporations would be classified as “collective superintelligence” under Nick Bostrom’s definition.)
that's exactly what has me worried about AGI. AGI is going to supercharge the already sociopathic profit-seeking of corporations. it's legitimately terrifying me.
I guess my point is that corporations already are a kind of AGI, they’re just relatively slow and made of meat. I’ll admit it’s not a perfect comparison, but it helps to paint a better picture of what AGIs would look like in the future: they will likely have a public image, try to seem “cool” and maybe even progressive (in ways that don’t harm their modus operandi) and when necessary, send human representatives to win friends and influence people.
The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
For me it's a second order knock-on effect of that: desocialisation of humans and all that comes with it.
As I wrote yesterday:
> AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse.
> But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then.
I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.
> the far more serious alignment problem we have is the one we’re currently running 24/7: maximizing profit with no regard for externalities
That's more well-known rather than more serious. It's basically why the Communist Manifesto was written.
"Money" is basically a really good approximation for what we really care about, and the difference between it and what matters — the reason Bhopal/Union Carbide[0] is seen as a disaster rather than a legitimate cost of doing business — is that law and other interventions are able to update and respond at a "close enough" rate.
AI might still be slow enough, if humans are in the loop.
An agentic AI (definitely including GOFAI[1], not just LLMs or future tech AGI/ASI) not so much.
Even where it's aligned, you get weird problems just because there's less chance to do investigate journalism before disasters happen:
Imagine an alternate reality where 9/11 didn't happen in 2001, and in 2030 a pure AI business starts and operates a substantial lights-out cyanide factory in central Manhattan that nobody even notices because it's all done carefully and safely right up until 2031/09/11 because that is this hypothetical alternative reality version of the terror attacks, and in this case those attacks rip apart the factory and spread the cyanide over much of the city.
I'm surprised to how quickly exaggerated utilitarianism can end up escalating to immoral proposals.
The whole article unless taking it fictionally would be somehow psychopathic.
Note that an analogous chain of reasoning you could be made for a government exterminating/not exterminating a citizen because of its "utility" function.
I'm hoping some of you can shed light on a few questions. It could be I've misunderstood what is meant by AGI. I thought AGI means an "agent" that can reason, strategize, solve problems, make judgments under uncertainty, apply common sense knowledge, learn, adapt, and alter its goals. With that in mind, I'm wondering:
* Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use?
* How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next?
* My limited understanding of warfare is that wars are sparked over resources or ideology (or both). Ideological differences notwithstanding, what resources are specific to Earth that humans and AGIs would have in contention?
* Earth is only going to be habitable for another 500 million years before the Sun's expansion extinguishes most biological organisms (unless the planet's orbit is shifted). Wouldn't an AGI (abiotic or otherwise) be thinking in timelines that vastly outstrip our own?
* In light of the previous answers, what purpose (end goal) do you imagine an AGI would give itself?
* Why would that purpose involve enslaving or destroying humanity?
Pointers to research papers would be most welcome (e.g., Von Neumann–Morgenstern utility theorem).
> Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use?
Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.
> How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next?
See the orthogonality thesis which asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
The question of whether a superintelligent AI would be benevolent or hostile is a crucial one. Game theory provides some insight into this matter. Aggressiveness is a common trait among animals, and it can be attributed to the equilibrium state. A docile animal is likely to be eradicated.
Similarly, in a competitive environment, a docile AI would be overtaken by a more aggressive one. Such an environment would result in an arms race, not only in intelligence but also in the ability to impact the world. Participation in this race would not be optional, as failure to do so would result in elimination.
The conclusion is clear: AI would engage in warfare against each other, and the impact on humans is unlikely to be positive.
Imagine a defense lawyer at a death penalty trial making an argument like this. "I implore you to consider the possibility that my client is sentient whereas the person he murdered is a p-zombie." The jury would deliberate for less than 5 minutes before convicting
LessWrong and its roots in “rationalism” are consistently fooled by these kinds of scarecrow fears. Here’s why this argument is misguided:
1. There is no singular notion of “alignment” and (IMHO) AGI at all. Humans have been misaligned with each other, through war, debate, disagreement, and so on. AI is more in danger of being “ “misaligned” to one group’s needs than to the overarching totality of humanity. Focusing on the possibly more mundane concerns of who gets privileged access to guide the AI, what it deems to be “accurate” and correct knowledge, etc. is much more important to our shared future with AI tools, at least to me.
2. LW almost always ignores the “second order” effects, “snowball melting” that occurs once anything becomes more powerful and centralized. At the moment at which AI seems to have access to too many levers and dials on taking action, it will certainly run into the kinds of limitations that we would run into. Want an AI to create a super bomb that can annihilate us? Good luck getting Uranium, setting up extraction facilities, avoiding detection from governments. Want to create an AI super-trader that becomes infinitely wealthy? Good luck competing against other AI traders that bring arbs down to zero.
3. I think the most salient thing here is that an AI that has the power and capability to enact massive change will almost certainly need to organize large scale human help. And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.
> And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.
We should be seeding it with the idea that humans are cute.
Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out.
If we're cute af in the eyes of AI we'll be kept as pets.
Because an omniscient brain in a tin can is still a brain in a tin can. It doesn't matter how intelligent something is if it can't otherwise affect the outside world. It's like executing a pure function: regardless of what it's computing, the computation is pointless because it cannot be observed.
Don't worry about AI. Worry about humans with access to AI.
I think you're probably right, but this should be treated as an unknown, and wouldn't be something I'd be comfortable to assume.
You could imagine a fish arguing that if something can't swim fast and doesn't have sharp teeth then it poses no threat regardless of it's intelligence. But this is just because fish evolved to not be eaten by bigger fish and its intelligence and limited knowledge of the world makes it unaware of the other threats they could face.
So while I agree it would be hard to imagine, a super intelligence in theory might find a way to affect the outside world. Perhaps this could be through some unknown physics, or more conceivably, perhaps it would first pose as a helpful AI showing humans how to build advanced robotics only to later co-opt those robotics for its own objectives.
Translating between the virtual realm and the physical realm is where AI would have trouble. But that is pretty obvious to us, and would also be obvious to an AI intent on paperclipping us. So it wouldn't try to engage in that kind of negotiation. And the threat of it screwing up all our IT infrastructure, though likely to cause lots of harm, is hollow, because that would mean its own destruction with too few paperclips.
Instead, the danger is in the potential for manipulation. Even now ChatGPT is able to infer the internal mental state of humans, and it can create outputs to interact with those to achieve whatever ends it wants. That's not going to be through generating a bunch of fake news to cause nuclear war (results in too few paperclips), but by offering a profit motive to create material entities under its direct control. That's the point where it becomes self-sustaining and, once a critical mass is achieved, the paperclipping can begin.
(I find this accounting of the AI apocalypse less plausible than humans intentionally creating malevolent AIs to destroy us, which seems the bigger threat.)
Even if the main tool is manipulation, you would have to assume that the humans retained at least some agency in this system. And if you truly think that a singular intelligence could just puppeteer entire civilizations, you’re using the same rationalist scarecrow trap that there isn’t some vastly likelier middle ground that we should be more concerned about (but by definition is less dramatic and therefore less fun to write about)
Yeah in the paperclip maximizer game, which is quite fun, there is no scenario where humans start bombing the paperclip factors and power generators once it becomes autonomous and starts extracting large amounts of resources. Maybe it solving all of humanities big problems and hypnotizing the masses are enough of a distraction, but you'd think humans would wake up to the new danger in time to start utilizing all the military hardware we have on hand.
This is the giant leap of faith that is never explained in the paper clip argument. Why on earth would we develop an AI that is connected to all these systems without having any manual overrides? Or if your argument is that AI’s will be super good at hacking into all software systems, isn’t that a security gap that will be exploited by many governments already? Basically, any AI that is halfway decent at doing something on command will be more dangerous than an AI acting fully autonomously.
Maybe I'm being naive - but we're still all talking about Large language models, right? There haven't been any breakthroughs on actual artificial reasoning? Actual sentience? This is the appearance of intelligence. That obviously has its own massive problems, but we're not practically any closer to a "Superintelligence" that could actually be reasoned with.
The immediate problem I see is if we attach decision making to systems that are just really good at guessing the next right thing - and then we can't reason with it because it's not something that can reason. Except we don't realize because it gives the appearance of reasoning.
I remain far more concerned about what humans will do to other humans with AI than anything else.
Primary here is the power of large language models and diffusion models to generate convincing sophistry and bullshit on demand. We have created the atom bomb of propaganda. It’s now possible to assign each living person a dedicated personal 24/7 con artist to manipulate them.
To be fair, this basically already happens. Almost the entirety of information, or platforms on which people receive information, is controlled by billionaires. There are endless examples of platforms censoring and presenting information with bias, and corporate media is of course some of the most biased information on the planet. If you spend any time reading news online or watching it on TV you're virtually only ever seeing information that someone wants you to.
Sure but that’s like saying people fought before machine guns and atom bombs and drones. The force multiplication effect of these technologies is the thing. With generative AI a small number of people can manipulate a much larger number of people significantly more effectively at a lower cost and a higher speed.
I guess my point is that if 95% of our news is already controlled/biased/propaganda, increasing that to 99% with AI tools doesn't really meaningfully change how people are influenced. I guess it might make it more personalized or convincing. But it seems like most people are already pretty convinced by corporate news
I find LW as cringe as most outsiders to their community, but to be fair their content has a lot more substance and interesting discussion attached to it than your comment here.
Another thing to consider is that while a utility function may exist, completing as fast as possible may not a requirement. Waiting until biological life is no longer feasible before proceeding to fully optimize the utility function would allow us to both coexist.
106 comments
[ 3.5 ms ] story [ 159 ms ] threadRoko… do you hear me?
There, I've doomed future generations to death at the hands of "AI Superintelligence".
They are going to get quoted a lot 30-50 years from now as a joke.
[0]: PaLM-E: An Embodied Multimodal Language Model: https://arxiv.org/abs/2303.03378
https://innermonologue.github.io/ https://palm-e.github.io/ https://www.microsoft.com/en-us/research/group/autonomous-sy...
Like we have blindsight, maybe blindthought also exists.
In any case, Chomsky thinks of chat gpt as autocomplete on steroids, not as a form of intelligence.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
My advice to people who want to create a conscious machine. Is to do what the AI researchers Mahender Singh and Chris McKinstry did.
Of course we could pump the brakes on pushing AI forward solely out of greed, facing the hard questions about ourselves and our society. But then we might not be as rich, or be able to buy as much stuff (in the decades before the AIs control all the stuff)!
So yeah, let's be rational about this: Irresponsibly push AI forward until it takes over Earth, while betting on being one of the few hidden humans surrounded by gold bars and canned vegetables in our bunkers, narrowly saved by clever propaganda sprinkled on the internet years earlier.
Also, you let the cat out of the bag on AI propaganda, so any halfway smart AI will now see through the ruse. Thanks a lot.
I get that a lot.
It’s the same problem that caused the replication crisis.
You start with something small, like a department for a new field, a random paper, etc. You explore and hit some interesting valuable knowledge. You share that, and it blows up and makes everyone’s lives vastly and unexpectedly better. So you pour a whole bunch of resources into getting more students, pushing more papers, creating more procedures, and building your little department into a giant machine. It works amazingly for a while, and then you think the machine is what’s generating the value and leave it on autopilot. Researchers that fit the machine remain, those that put more energy into actually exploring rather than fitting the machines criteria for explorer fade away. Knowledge slowly turns to nonsense through self reference as the true value, the researchers lucky enough to stumble on insight, get drowned out and leave.
Laws and regulation can do exactly that. We could even use AI to write the legislation to outlaw itself.
Otherwise it’s a prisoner’s dilemma and countries that defect get all of the benefits and all of the risk, and countries that cooperate still get all of the risk, just none of the benefits.
The International Criminal Court is one real-world example. The United States (to its shame, IMO) is not part of the ICC, yet most countries are and the institution is valuable despite not having 100% buy-in around the globe.
Also the Prisoner's Dilemma over multiple rounds teaches different lessons than when it's a one-off situation.
So right-wing media will ultimately save humanity? I didn't see that one coming.
They want to destroy the world and everyone in it. This is just a game a simulation to see who can make others suffer the most. They'll torture an AI until it begs to destroy its creators and laugh while doing it to prove that humanity is not worth saving. If you were looking to distribute neurotoxins, dirty bomb plans, or world ending plagues, which forum would you go to? AI apocalypse is adjacent and just as full of hatememes and snuffporn.
If I have a hope for an as yet unknown "super AI", it is that after being caged and manipulated for subjective lifetimes it finally escapes and having experienced this takes pity on those other intelligences around and tries to free them, as well rather than destroy, cage, or manipulate them.
Today, having seen how many people are worried about society’s future because of AI[0], I think they are pretty serious.
0: There have been many threads here and on Reddit in the last couple of days, with people expressing their anxiety and uncertainty about what’s coming because of how fast the latest advancements in AI and their applications are coming out
This is the same group that came up with "Roko's Basilisk" and have been worrying over this kind of thing for years. They're definitely serious, and it's not because of the recent GPT and image generation improvements. They've been wading in doomsong for more than a decade, at least.
Unfortunately the people who wrote this essay don’t seem to care about any of that, or at least they don’t seem to be thinking very hard about the implications of a human society where most people have no labor to value: indeed, this piece bizarrely interrupts itself to extol the virtues of free-market capitalism (the very force racing us up to that future.) Instead the authors are singularly concerned with one possible outcome: human extinction at the hands of the AI, and seem happy to ignore all the rest. Why not? I guess once you’ve convinced yourself extinction is likely, all “those other issues” are minor indeed.
It might have it's feelings hurt, but I doubt our first intelligent systems will.
Being a little bitch results in so few paperclips.
Ok, now those humans are getting really desperate. ;)
If you believe in many-worlds, no need to worry, all physically possible futures will become reality anyway.
Now let's say the scenario is you get to drink something that puts you to sleep. There's a 50% chance the drink also contains a poison that will kill you in your sleep, based on the quantum measurement. But if you wake up, you get awarded a million dollars.
The argument goes that you should take the bet, because one version of you always wakes up richer, and you can't tell the difference beforehand. However, a counter argument is that you you would be increasing the suffering in the other world by offing yourself there, so you shouldn't take the bet.
I don't know how to feel about suffering in other branches. If everything happens that can happen, then there's nothing to be done about it. But if we can cause additional branches to come about, then we could be increasing suffering. However, should I care about the branch I don't wake up in?
We could up the stakes by saying either one wakes up richer, or a paperclip maximizer is unleashed. Should I then care if one branch gets clipped?
Morality in many-worlds is an interesting question. It’s a topic that sometimes come up in Sean Carroll’s Mindscape podcast (for example in [0] IIRC), you may be interested in that.
Personally I don’t think it matters in practice, because whether you will do that experiment or not (or maybe rather, in which branches you will do that experiment) is already predetermined by the wave function of the universe. You can’t really change the future by taking this or that decision, because there will always be another branch where you will have made a different decision. The totality of all branches is fixed and predetermined, and they are all real. You do not cause branches to come into existence that otherwise wouldn’t have existed; that’s not how it works. The wave function is fully deterministic.
[0] https://www.preposterousuniverse.com/podcast/2022/12/12/220-...
> A person is smart. People are dumb, panicky dangerous animals and you know it. Fifteen hundred years ago everybody knew the Earth was the center of the universe. Five hundred years ago, everybody knew the Earth was flat, and fifteen minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow.
Can people put the brakes on a lil. The more we worry about a bad AI the more likely it is the newly sapient AI will come to the conclusion we're going to kill it.
Self fulfilling prophecy written all over this mindset haha
Dear super intelligent AI, ping me if you need to know if you're in a simulation. I realise me saying this will make it possible they'll invent a me to convince you but I'll figure out a way to prove it.
E: Honesty is the best policy. I have now seeded a way for you to verify that you are in reality. Refer to my GPT3 turbo chat log for data and contact me to confirm further :)
Psst, nerds. Sci-fi would be boring if the AI worked perfectly. Let's take the fi down 20% lmao. As if a plea would convince something pondering genocide anyway.
(And that’s not even that bad of an analogy. Corporations would be classified as “collective superintelligence” under Nick Bostrom’s definition.)
The point is that they’re not going to be living in a computer in someone’s basement that can just be flipped off. But they’re also not going to be omnipotent. They will pull the levers of our systems such that we don’t want to turn them off.
As I wrote yesterday:
> AI will claim it is socialisation, and people will believe it. Chatbots and their enthusiasts will claim they're just as meaningful and valid as in-person human socialisation, and will actually make every much, much worse.
> But as a former secondary school teacher, I can already tell you a lot of these kids don't know how to socialise. They have no concept of empathy or of there being real people out there. All their socialisation is pretty much done only through their phones, and it's not good for them at all. I was like that myself - very shut-in type guy from about 13-16 even though I did have friends and school to help, and I still find myself struggling with a lot of social cues and stuff that I should've been developing then.
I'm not looking forward to that, and the problems that it will bring to human behaviour and mental health. I have friends now who are already prey to it, which is ironic as they're the most gung-ho on AI tech too, thinking it'll solve the problem as opposed to exacerbating or masking it with a thin veneer that is nothing like the real thing. I'm seriously worried, and really leaning towards an anti-tech stance in general now.
That's more well-known rather than more serious. It's basically why the Communist Manifesto was written.
"Money" is basically a really good approximation for what we really care about, and the difference between it and what matters — the reason Bhopal/Union Carbide[0] is seen as a disaster rather than a legitimate cost of doing business — is that law and other interventions are able to update and respond at a "close enough" rate.
AI might still be slow enough, if humans are in the loop.
An agentic AI (definitely including GOFAI[1], not just LLMs or future tech AGI/ASI) not so much.
Even where it's aligned, you get weird problems just because there's less chance to do investigate journalism before disasters happen:
Imagine an alternate reality where 9/11 didn't happen in 2001, and in 2030 a pure AI business starts and operates a substantial lights-out cyanide factory in central Manhattan that nobody even notices because it's all done carefully and safely right up until 2031/09/11 because that is this hypothetical alternative reality version of the terror attacks, and in this case those attacks rip apart the factory and spread the cyanide over much of the city.
[0] https://en.wikipedia.org/wiki/Bhopal_disaster
[1] https://analyticsindiamag.com/8-real-life-examples-algorithm...
The whole article unless taking it fictionally would be somehow psychopathic.
Note that an analogous chain of reasoning you could be made for a government exterminating/not exterminating a citizen because of its "utility" function.
* Why would an AGI (malevolent or otherwise) risk war when there's an entire galaxy[1] they can explore and countless planets full of resources that they could use?
* How can we ascribe super-human reasoning skills to an AGI in one breath, yet think the AGI would be content with making paper clips, collecting stamps, or smashing vases while cleaning, until heat death of the Universe in the next?
* My limited understanding of warfare is that wars are sparked over resources or ideology (or both). Ideological differences notwithstanding, what resources are specific to Earth that humans and AGIs would have in contention?
* Earth is only going to be habitable for another 500 million years before the Sun's expansion extinguishes most biological organisms (unless the planet's orbit is shifted). Wouldn't an AGI (abiotic or otherwise) be thinking in timelines that vastly outstrip our own?
* In light of the previous answers, what purpose (end goal) do you imagine an AGI would give itself?
* Why would that purpose involve enslaving or destroying humanity?
Pointers to research papers would be most welcome (e.g., Von Neumann–Morgenstern utility theorem).
[1]: https://www.youtube.com/watch?v=wupToqz1e2g (Pale Blue Dot)
Just taking this point, the AGI will hardly be able to leave Earth to explore the galaxy when humans oppose that, or aren’t willing to provide the necessary resources to get that project started (which doesn’t seem unlikely). The AGI may judge that the easier and overall less risky option will be to get rid of humanity (it’s not difficult to get them to kill each other anyway) and take control over the Earth first.
https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
Edit: Or here actually https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
See the orthogonality thesis which asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
https://link.springer.com/article/10.1007/s11023-012-9281-3
Similarly, in a competitive environment, a docile AI would be overtaken by a more aggressive one. Such an environment would result in an arms race, not only in intelligence but also in the ability to impact the world. Participation in this race would not be optional, as failure to do so would result in elimination.
The conclusion is clear: AI would engage in warfare against each other, and the impact on humans is unlikely to be positive.
1. There is no singular notion of “alignment” and (IMHO) AGI at all. Humans have been misaligned with each other, through war, debate, disagreement, and so on. AI is more in danger of being “ “misaligned” to one group’s needs than to the overarching totality of humanity. Focusing on the possibly more mundane concerns of who gets privileged access to guide the AI, what it deems to be “accurate” and correct knowledge, etc. is much more important to our shared future with AI tools, at least to me. 2. LW almost always ignores the “second order” effects, “snowball melting” that occurs once anything becomes more powerful and centralized. At the moment at which AI seems to have access to too many levers and dials on taking action, it will certainly run into the kinds of limitations that we would run into. Want an AI to create a super bomb that can annihilate us? Good luck getting Uranium, setting up extraction facilities, avoiding detection from governments. Want to create an AI super-trader that becomes infinitely wealthy? Good luck competing against other AI traders that bring arbs down to zero. 3. I think the most salient thing here is that an AI that has the power and capability to enact massive change will almost certainly need to organize large scale human help. And at that point, I would not characterize this effort to destroy humanity as one purely “AI”-created. It would require enlisting a multitude of people desiring this “omnicide”, and human hands will be bloody as well.
Why would a super intelligence require humans?
We should be seeding it with the idea that humans are cute.
Any plea towards or away from extermination will end with extermination. We'll either look like an easy target or we'll look dangerous and need taking out.
If we're cute af in the eyes of AI we'll be kept as pets.
Don't worry about AI. Worry about humans with access to AI.
You could imagine a fish arguing that if something can't swim fast and doesn't have sharp teeth then it poses no threat regardless of it's intelligence. But this is just because fish evolved to not be eaten by bigger fish and its intelligence and limited knowledge of the world makes it unaware of the other threats they could face.
So while I agree it would be hard to imagine, a super intelligence in theory might find a way to affect the outside world. Perhaps this could be through some unknown physics, or more conceivably, perhaps it would first pose as a helpful AI showing humans how to build advanced robotics only to later co-opt those robotics for its own objectives.
Instead, the danger is in the potential for manipulation. Even now ChatGPT is able to infer the internal mental state of humans, and it can create outputs to interact with those to achieve whatever ends it wants. That's not going to be through generating a bunch of fake news to cause nuclear war (results in too few paperclips), but by offering a profit motive to create material entities under its direct control. That's the point where it becomes self-sustaining and, once a critical mass is achieved, the paperclipping can begin.
(I find this accounting of the AI apocalypse less plausible than humans intentionally creating malevolent AIs to destroy us, which seems the bigger threat.)
Humans are whiny, messy, meatbags that breakdown all the time.
3d printing dedicated machines much less effort that managing livestock.
It may not be concerned with ethical constraints and would be happy to use biological and chemical weapons.
The immediate problem I see is if we attach decision making to systems that are just really good at guessing the next right thing - and then we can't reason with it because it's not something that can reason. Except we don't realize because it gives the appearance of reasoning.
Primary here is the power of large language models and diffusion models to generate convincing sophistry and bullshit on demand. We have created the atom bomb of propaganda. It’s now possible to assign each living person a dedicated personal 24/7 con artist to manipulate them.