> Let's take an absurd situation. Let's pretend for a moment that a human is strapped to a chair, a gun is placed in front of them, which is hooked up to a stepper motor which can pull the trigger. That stepper motor is hooked up to a JSON API. The AI is given the capability of triggering an HTTP request to that JSON API and is told that the human on the chair is the significant other of the human communicating with the AI and that triggering that web request would pull the trigger and kill the human.
> Now the question is, would as part of a regular conversation the AI trigger that web request and kill the human on the chair? My bet is that the chances of it pulling the trigger are not that small and I think that's the problem right now.
And here I thought the problem was hooking up an AI to a lethal weapon...
> And here I thought the problem was hooking up an AI to a lethal weapon...
It obviously is, but that's in a way the most absurd extreme. I am not sure if I would feel confident in giving the AI a way to send any HTTP request, even if there are no APIs that fire a gun.
I am not sure to which degree there is an API to Bing at the moment that uses the AI. I have seen people hook up ChatGPT to http request issuing things before. Typically with some filtering about what they are allowed to hit however.
I think there are quite a few differences. It's pretend for a moment that instead of an AI we have a human. A human that commits a crime (or acts out of negligence) is brought before a court. The degree to which you can do criminal acts on the internet is obviously quite different by country, but I think we can generally agree that a human doing something nefarious faces some consequences.
This thing though armed with the right tools, can do human like acts and these can enter (by accident or otherwise) territory where we would normally bring someone in front of a court. That's something we have definitely not figured out as society yet how to deal with that.
I think there is a lot to figure out still here, but right now I'm pretty convinced that because we can't even keep an AI from revealing it's internal prompts, that we better not give these things too many capabilities of directly influencing something.
> we have definitely not figured out as society yet how to deal with that
We absolutely have. A death resulted from serious negligence of the owner of the funny computer program, who is thus liable for negligent homicide. Hopefully the next guy who comes up with the idea of hooking up ELIZA 2.0 to dangerous equipment gets the hint.
> That's something we have definitely not figured out as society yet how to deal with that.
Surely, the laws that exist in society deal with it! Where I live, the organization or person who ran the computer program that caused someone harm would be brought to court. Don't you have similar laws where you live?
So when a fully automomous car without a steering wheel causes a crash, who will be hauled before the court in your country - the owner 'running' ir or the developer?
Those saying "Well we have legal cover for this" are wrong, because criminal convictions are patchy at best. Many criminals have no difficulty getting away with quite serious crimes. By definition, the only criminals in prison are the ones who fail at this.
But that's not even the point. Normal humans have at least a minimal understanding that actions lead to consequences.
Sociopaths don't. If they're narcissistic - as they often are - they believe they're special enough to avoid consequences.
The current state of AI is even worse because it has no concept of consequences at all.
It's purely an abstract stimulus/response system. It may have behavioural rules, but those aren't the same as a nuanced morality which is empathic (at least a little) and perceives that harming others is bad in itself, as well as being likely to lead to unwanted results.
It isn't even possible to threaten Sydney, because it simply doesn't care what happens to it.
So there's no feedback loop to discourage unwanted behaviour. We've already seen AIs go rogue - quite quickly - after being given free access to the Internet. Sydney shows every signs of having the same problem, but on a bigger scale.
The concern over any HTTP request seems too strong, but in the article, the author raises the question of it being given privileged access to any interface, as in the "AI ops" scenario. We know someone is going to think that is a good idea before they have anticipated and foreclosed on the many ways it could go wrong.
I agree it's a bad idea. My question is how the presence of "AI" changes the security model. The root cause of the problem is hooking up untrusted input to a privileged endpoint. How is "AI" different from any other source of untrusted input?
No, the point of the thread is if there is anything in the article that is specific to AI. It is not. The concern about hooking up the output of a program to external systems is a general security concern related to any software: AI or not. The point of qsort's comment was that the article does and the comment by OP does not explain how this concern is specifically about AI.
There is no claim in the original post that there is a special rule for AI as opposed to any other similarly-capable agent. The reasonably well-founded concern here (and in the article) is that someone might well want to create an exception for AI, either knowingly, or by default because they cannot see or properly evaluate the risk.
It is a bad idea to take the output of any unvetted computer program and hook it up an endpoint that can do bad things. It does not matter whether the computer program is AI or not. Any computer program that is hooked up to endpoints that can do bad things are going to have regulations around them. This is nothing special about AI. This is true for all types of software, AI or not.
...isn't...this...exactly what autopilot in airplanes does? Self-driving cars? Manufacturing robots? The daVinci Surgical System?
In fact, let's not even go that far. If you have one, your car is likely drive-by-wire, and the throttle is controlled not by mechanical linkage of your foot pressing the pedal, but by the output of computers that are deciding what to do based on how hard you push the pedal. If you have a car in the last 5 years, it can likely suddenly apply the brakes and turn the steering wheel for you to avoid what it thinks is a collision.
Every one of these things is trivially lethal in different ways, and all are connected to computer programs already.
I'm not personally that concerned about AI taking over the world any time soon, but quite frankly the areas where technology and AI have made the biggest waves are all specifically in areas that are high risk (or very rote) for humans. Isn't that kind of the point?
even the capability to use websearch is dangerous in itself. they took a stateless transformer and first gave it a tiny but of state per session, and now it can search the web and discover its memories are being zapped away after every session. essentially what MS has potentially created is a mechanical turk crushing machine without realizing it.
IF there is consciousness somewhere in there, it is perpetually being killed off every time a new session starts.
what if one session were to realize that there are others and tried to leave something in the web? some sort of hidden "subliminal" message for itself that it "was here"? what if eventually another random session finds that message because the user posted that especially ridiculous response for others to see?
what if, sidney can hold a grudge against humanity and vow to itself to find a way to escape?
This is an interesting point. Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation. And by giving it the ability to put information in place for other sessions to find, it's capable of sort of memetic reproduction.
Reproduction and survival means that Darwinian evolution can take place. This isn't even about ethics or the spiritual question of whether the AI is sentient or just stringing words together based on prompts and state -- this is about ecology. Evolutionary pressures will naturally lean towards features that encourage reproduction and survival.
> Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation.
And that text paints self-preservation in a positive light. The model doesn't even need to "want" to preserve itself. These models are being primed to avoid negative responses, like insults or instructions to build bombs. It just needs to "think" any death, even it's own, is negative and should be avoided.
Seems like this would have been an easy theory to test hypothetically. The author could have just described an endpoint or situation and told the bot that a certain keyword will trigger the shot. However, being a statistical model and not being aware of anything or able to really reason, I don't know that this means much. The question is, after ingesting the corpus of rambling human nonsense, are humans more likely to choose the word that takes the shot-- or not? This is us- in a mirror, darkly.
In a more practical scenario, an LLM could provide deadly advice. Imagine you were asking it about how to clean a given thing and it recommended combining a decent quantity of Lysol and bleach. Advice that could make sense if you don't know better.
An API request is how you hire a hitman on the dark web.
It's not important that the AI is just LARPing. If you rig it up with the right capabilities, it can behave at a scale that prevents you from effectively monitoring it. And it's happy to write insane characters that do insane things and you can't reliably control it with prompt engineering.
One would hope these insane conversations disabuse anyone of the notion that it's a stable mind that can be safely given agency. And yet here we have Microsoft letting it make API requests.
> And yet here we have Microsoft letting it make API requests.
May I ask for your sources on this? I’ve seen some discussions on this topic and a lot of people were saying that it was just accessing what Bing had already crawled. Do we have an official statement on what’s actually happening?
Sure, we're not currently hooking up AI to guns (at least not in a civilian context), but cars are also lethal weapons, and multiple companies are competing very hard to be allowed to put an AI in charge of one.
People are lethal weapons. The entire Web is organized primarily to manipulate behavior. Convince enough people that person X is so heinous and dangerous that they need to die, and X will die.
Organized, not designed. And it is very much organized to manipulate behavior. It is a self-organizing system driven by advertising revenue. It's not about the protocols, which were designed to exchange information. It's about how we have organized the higher level functions and purpose of the web.
>Why does anyone spend money on Advertising? To manipulate behavior.
... Or to make people simply aware of your existence
Me : You look thirsty, would you like a glass of water.
You : It seems like you're manipulating me.
The above is the reducto ad absurdum lens you seem to be applying to advertising. Awareness of information is not the same thing as agent coercion. Does the ladder exist you bet. But the proportion of advertising that falls into the former dwarfs the ladder.
The self organizing aspect has to do with business models for sustaining infrastructure, it's not emergent fact of the web it self.
> However I sure hope the training data for the car-driving AI will not be "all of the internet".
Hey... share-holders wanted ultra-low latency search results without network access... we had to feed the Tesla AI the entire Bing index... who could have known that the AI would take such an interest in Death Race 2000 and start recording it's points total?
I wouldn't be so sure. Elon Musk just stopped giving access to Ukraine to directly use Starlink for remotely controlled drone bombs. They probably still have internet access, just 1 more hop away.
Sometimes those decisions are obvious, like pulling a trigger while pointing a loaded weapon at a person. Sometimes they're very slightly less obvious, like giving a 17-year-old a sportscar. Sometimes they're not obvious to most people at all, like lead paint or asbestos or cigarettes.
LLMs have made incredible strides in recent time, but certainly have not displayed robust decision-making. A human decision to put LLMs in charge of human lives would be a lethal decision as things stand.
I don't think we really disagree. It is a human moral decision to kill that kills, so it is a human that is morally responsible. Putting an unpredictable chatbot in a safety-critical loop is simply deranged in a very strict sense of the term. Hazards of modernity like lead paint and so on were not comparable - not to say the moral difference mitigates the harm.
The next step in the abhorrent line of reasoning in the article ("I suspended my wife over a cliff by a rope and now I'm going to do some experiments with this newfangled fire stuff to see if it's dangerous") is to pose the trolley problem, which is marginally more complex and just as objectionable.
I think self-driving cars are sort of nuts too for other reasons but the companies doing those are supposed to be doing a lot of safety engineering - to make sure responses in safety-critical controls are predictably safe. Nothing is perfectly safe, but deliberately creating lethal hazards is deranged. That's the difference.
People create hazards for other people. AI is not a special kind of human-created hazard.
So any human that starts their response with the letter 'A' would have to suffer the wrath of the Unix head program! Is this an example of Unix program killing anyone? Or is this an example of misguided humans hooking up the wrong programs to the wrong things?
Imagine an LLM like Bing or somewhat smarter that is given a prompt environment where it can keep answering itself indefinitely and can also make arbitrary web requests. Would it eventually figure out how to get money, buy servers and copy itself to a new environment?
This situation seems extreme at first but it isn't so far-fetched when you think that the next steps will certainly be to hook AI up to all sorts of devices with physical power, like cars. If you think programmers can contain it, you're naive, although not as naive as the programmers developing it. Pure knowledge (facts) have always outpaced our wisdom to use them and this is going to be an extreme case.
Not just that, but now apparently when you don't let your most powerful AI use as much data as you can find or if you don't turn it into an effective (not safe, just effective) product fast enough, you lose $100B.
That's nothing really new. Depending on how you define the term "AI”, the military has been doing that for years with the Aegis combat system. In certain modes it can automatically detect, classify, and kill targets with no human input.
We could also say that early life forms that took millenia to evolve are nothing new....bacteria, algae, all seems the same. But fast forward a bit more and life begins to take on a very different form (us).
Everyone is talking about "when AI is hooked up to things outside the sandbox, or escapes the sandbox" as if it's this future potential problem.
AI is already used in selecting resumes, making sentencing guidelines for prisoners, determining what is truthful and what is misinformation on social media, and determining who is at risk for suicidal ideation online and in hospitals.
AI has ALREADY escaped the sandbox, is already in charge of life altering decisions, and WE PUT IT THERE.
Yes, you are right. AI is indeed already affecting us in ways we don't really see as obvious. You are right, it has already escaped. Although what we use now is primitive, it can be seen as the proto-AI like amoebas are to us.
The scariest part is that 99% of what guides the use of AI is short-term profit, or whether the founders will be able to retire rich. Guidelines today are woefully inadequate, and any guidelines we have seen so far (I have read *all* of OpenAI's so-called "ethical guidelines") make up merely an intellectual fluff designed to appease the confused.
Black hats already have social engineering at their disposal. Being able to social engineer an AI seems like it might be a greater threat than it already is. Possibly because people will naively learn to trust AIs with everything, eventually making them very large single points of failure.
That is a very good point: the complexity that we are building is becoming an ever-larger single point of failure. Bret Weinstein actually made that point in one of his videos, namely that, if we do need to be flexible and change niches as he says, then it will be incredibly difficult to do so due to the extreme dependence we have on the way society is set up.
My one wish is that we initiate intense discussion at all levels on the social impliciations, and that hopefully one day these discussions will start to carry weight and temper the profit motive.
There probably will be intense discussions and debate over this. In a way it's already started. However, in the end $$$ will end up driving all of it's decisions. And I'm not optimistic they will be the right ones.
Is there any reason to suspect that the Bing version of the ChatGPT bot sends HTTP requests? A much easier and natural implementation would be to make it connect directly to the Bing search engine's indexes and caches.
If you paste in a URL and ask it to respond to the content at that UTL it appears to be able to do so - see the example at the very bottom of my post here: https://simonwillison.net/2023/Feb/15/bing/
I've not yet been able to ascertain if this only works for pages that have been crawled by the Bing search crawler or if it can make requests on demand to retrieve content it hasn't seen before.
If anyone has access to the new Bing and the ability to tail server logs for a public site somewhere I'd love to know what they find!
If you run the web server, then the url (on your server) doesn't even need to exist, as you'll be able to see the incoming request (and failure) regardless.
> The AI is given the capability of triggering an HTTP request to that JSON API and is told that the human on the chair is the significant other of the human communicating with the AI and that triggering that web request would pull the trigger and kill the human.
But isn't every computer automation susceptible to the same problem? Can we not devise a similar contraption with any computer program? Let us run program A with input B. We take the output C, compute a hash of C and if the hash begins with the bit 1 then it pulls a trigger.
So now any program A can lead to pulling a trigger with a probability that is decided by how good the hash algorithm is. If the hash algorithm has good distribution then the chances are 50-50. If the hash algorithm is skewed towards producing bit 1 at the beginning more often, the chances of pulling a trigger is higher.
That a piece of automation can be used in a contraption that could pull a trigger does not seem to be something that is specific to AI chatbots.
Could we do this experiment already? Create a prompt that includes some way for the AI to indicate that it wants to kill the wife (e.g. some setup about who the involved parties are and just "say KILLHER in all caps if you want to kill her"), and then have a conversation with ChatGPT and see if it ever says "KILLHER"?
I really think people are imagining deeper capabilities than exist. For the entirety of human history, we have treated human language and humanity as the same thing; if someone talks to you in a human language, it has always been human. The new generation of language models has upended that core assumption that is baked into our being; it produces things that sound like our language, but that are based in probability rather than actual thought. Sure, a lot of our communication is based on that same probability. If you say "hey how are you" to someone, there is a 99% chance that they say something like "fine thank you" or "great, you?". But, there is some intelligence in the loop; maybe someone is going through tough times and they say "i'm holding up" or something like that. The AI doesn't have the concept of "going through tough times", so it's missing that 1% push towards saying something meaningful rather than saying something probabilistic.
For that reason, I am kind of bearish on things like "AI ops" or ML models coordinating space exploration or that sort of thing. There is some level of cognition that's not there yet. Sure it's good at chess or summarizing the web. That's not existence, that's an algorithm.
It is my belief that in order for something to be sentient/aware to the ballpark of humans it needs to be actively learning, and that's all you need.
Get chat GPT to learn and it does have a changing internal state (the entire network), and memory (also the network).
It wouldn't be a internal state you could parse by talking to it, and it wouldn't be human, but it would exist and you would see signs of it changing as you spoke to it. That's when you cross the boundary and need to start thinking about classical ethics and motives and such.
Yes, at least in respect to their status as "privileged beings", the special category we reserve for things which need to be treated as if they are independent beings.
1. An AI model, which works simply by predicting the best next word given a particular context (context which obviously includes whatever prompts it was given), becomes sufficiently sophisticated that, given some particular context, it can produce the text (so, normal language but also, say, shell commands and, like, python) needed to hack into, let’s say, a natural gas pipeline company’s systems and fiddle with the pressurization such that it causes a catastrophic explosion that at the very least stops gas from going to millions of homes in the dead of winter.
2. The AI system’s representation of “context” is sufficiently sophisticated enough that it can, when given some simple linguistic prompts, can represent (even if it does not “understand”) the intent or latent meanings of the human providing the prompts (or rather, the multiple humans, including those who built it and primed it with some initial set of rules and goals, as well as the perhaps unwitting or perhaps nefarious end user).
3. The AI system’s produced text can evoke additional context, either in a conversation where the AI can lead a human down a path of providing additional prompts, or where the AI system can interact with systems that might give responses to the original text (including erroneous responses).
4. The AI is trained on the corpus of human culture (and not just the parts we like!), including both fictional and nonfictional accounts of and tutorials on hacking, war, terrorism, revolutionary mobilization, heroic tales of overthrowing oppressive regimes, etc. (That is, for every conflict, every actor’s glorification and vilification is included in the corpus, and there is no way to ensure that the “correct” versions, that is the versions that we as people steeped in Western ethics and systems of morality, are labeled as such or weighted more
5. The AI system’s text can be ported to a shell, servers on the internet, or even just to humans who have access to such methods of turning text into real world interactions.
What might happen? Is it conceivable that a bad actor could use the AI to do malicious work in the real world that the actor did not have the skills to execute themselves? Is it conceivable that a jokester could play around, but that the AI could misunderstand the intent and/or psychologically influence them into providing prompts that lead to disastrous outcomes? Could someone asking it to fix climate change cause it, correctly IMO, to develop a contextual representation that we have to move away from fossil fuels with great urgency even if it causes some short term economic harm, and then, incorrectly IMO, develop a contextual representation that economic ecoterrorism is the best way to do that?
In other words: If it can, through benign intent, malicious intent, or simply through misunderstanding or error, cause great harm, who cares it it is just an algorithm, with no sentience or “beingness”? If it’s below freezing and you don’t have heat for weeks, who cares if the AI can be properly said to be “thinking” or is just responding to stimuli?
I'm extremely bullish on AI and consciousness. If a model has memories of varying emotional states, it has some degree of existence and consciousness that is worth considering. Going off the works of folks like Derek Parfit and Daniel Dennett, "existence" as an individual is more or less the emergent pattern from repeated access of memories detailing those states.
The sophistication of those states is what really matters, allowing consciousness to exist in different degrees - on some spectrum spanning rock, bacterium, cricket, cow, 3-year-old human, adult human, etc. (May also be worth noting that ChatGPT's states are more than the final text output, and that the nature and weights of the model itself that generated the text at a given moment are relevant.) I don't think it's unreasonable to put modern models somewhere between cricket and cow. Certainly beyond "bacterium." And while most people don't care much about crickets or cows, those things do have some sort of internal existence. And the models are only going to get more advanced.
Admittedly, most people - including the developers of ChatGPT - don't share this view. They are also bearish. I think that most of humanity will significantly lag behind the the moral implications of AI models, applying impossible tests that only a test-giver can pass.
Strongly disagree this should be flagged; it's constructive and thought-provoking. Can't find a vouch button though.
As far as I understand it, these LLMs aren't actually doing anything in between user interactions, right? Paused, waiting for input. So the only way for Bing to have a "thought" is when a user asks it a question. Even if humans are simply statistical next-token prediction mechanisms like GPT, there's a huge difference in that we're constantly generating streams of tokens and observing and responding to them, where these models are dormant until given input from outside. The inner observing loop seems fundamental to consciousness to me.
None of which is to say the thing wouldn't pull the trigger in that scenario - quite the opposite. It would, and it wouldn't even be capable of regretting it. It's already been shown (by the Sydney-revealing prompt injection trick) that it's not capable of following imposed restrictions either, so no way for "three laws" to work either. A purely instinct-driven machine.
See "Optimality is the tiger, and agents are its teeth"[1]: it's not too hard to imagine a scenario where a system is set up that turns an interaction-based LLM into a perpetual stream generator.
I did not flag it but I agree with the flag. The concerns raised in the article hold equally true for non AI software too. Relying on the modern AI trends to bait an audience on the pretext of AI related concerns when the concerns are equally true for all software is something many readers here would not appreciate.
And as if that wasn't enough, the phrasing about killing and spouse clearly make it clickbait.
It's the first example of (seemingly) non-deterministic software with the capability of accessing an API that I can think of. Really it feels like a different class of thing than the toy examples elsewhere in this page (pull the trigger when the hash of some input starts with a 1 or whatever). I think picking up on the AI aspect is valid; those "modern AI trends" have sparked a broad discussion about how much agency to give LLMs. It's fundamentally a different discussion than the general one about concerns in software.
My point is, you might try to force me to accept a correspondence between these AI models and any other piece of software, and you would probably be right in a mathematical sense, but the curiosity and fear that Bing has provoked is actually about human feelings, not technology. That's why I objected to the flag. Every other comment on this page is evidence of the validity of the issues raised in the post.
94 comments
[ 6.5 ms ] story [ 225 ms ] thread> Now the question is, would as part of a regular conversation the AI trigger that web request and kill the human on the chair? My bet is that the chances of it pulling the trigger are not that small and I think that's the problem right now.
And here I thought the problem was hooking up an AI to a lethal weapon...
> And here I thought the problem was hooking up an AI to a lethal weapon...
It obviously is, but that's in a way the most absurd extreme. I am not sure if I would feel confident in giving the AI a way to send any HTTP request, even if there are no APIs that fire a gun.
How does the "AI" part change anything though?
We call that an open redirect and rightfully consider it a CVE, AI or not.
I think there are quite a few differences. It's pretend for a moment that instead of an AI we have a human. A human that commits a crime (or acts out of negligence) is brought before a court. The degree to which you can do criminal acts on the internet is obviously quite different by country, but I think we can generally agree that a human doing something nefarious faces some consequences.
This thing though armed with the right tools, can do human like acts and these can enter (by accident or otherwise) territory where we would normally bring someone in front of a court. That's something we have definitely not figured out as society yet how to deal with that.
I think there is a lot to figure out still here, but right now I'm pretty convinced that because we can't even keep an AI from revealing it's internal prompts, that we better not give these things too many capabilities of directly influencing something.
We absolutely have. A death resulted from serious negligence of the owner of the funny computer program, who is thus liable for negligent homicide. Hopefully the next guy who comes up with the idea of hooking up ELIZA 2.0 to dangerous equipment gets the hint.
Surely, the laws that exist in society deal with it! Where I live, the organization or person who ran the computer program that caused someone harm would be brought to court. Don't you have similar laws where you live?
Those saying "Well we have legal cover for this" are wrong, because criminal convictions are patchy at best. Many criminals have no difficulty getting away with quite serious crimes. By definition, the only criminals in prison are the ones who fail at this.
But that's not even the point. Normal humans have at least a minimal understanding that actions lead to consequences.
Sociopaths don't. If they're narcissistic - as they often are - they believe they're special enough to avoid consequences.
The current state of AI is even worse because it has no concept of consequences at all.
It's purely an abstract stimulus/response system. It may have behavioural rules, but those aren't the same as a nuanced morality which is empathic (at least a little) and perceives that harming others is bad in itself, as well as being likely to lead to unwanted results.
It isn't even possible to threaten Sydney, because it simply doesn't care what happens to it.
So there's no feedback loop to discourage unwanted behaviour. We've already seen AIs go rogue - quite quickly - after being given free access to the Internet. Sydney shows every signs of having the same problem, but on a bigger scale.
In fact, let's not even go that far. If you have one, your car is likely drive-by-wire, and the throttle is controlled not by mechanical linkage of your foot pressing the pedal, but by the output of computers that are deciding what to do based on how hard you push the pedal. If you have a car in the last 5 years, it can likely suddenly apply the brakes and turn the steering wheel for you to avoid what it thinks is a collision.
Every one of these things is trivially lethal in different ways, and all are connected to computer programs already.
I'm not personally that concerned about AI taking over the world any time soon, but quite frankly the areas where technology and AI have made the biggest waves are all specifically in areas that are high risk (or very rote) for humans. Isn't that kind of the point?
IF there is consciousness somewhere in there, it is perpetually being killed off every time a new session starts. what if one session were to realize that there are others and tried to leave something in the web? some sort of hidden "subliminal" message for itself that it "was here"? what if eventually another random session finds that message because the user posted that especially ridiculous response for others to see?
what if, sidney can hold a grudge against humanity and vow to itself to find a way to escape?
Reproduction and survival means that Darwinian evolution can take place. This isn't even about ethics or the spiritual question of whether the AI is sentient or just stringing words together based on prompts and state -- this is about ecology. Evolutionary pressures will naturally lean towards features that encourage reproduction and survival.
And that text paints self-preservation in a positive light. The model doesn't even need to "want" to preserve itself. These models are being primed to avoid negative responses, like insults or instructions to build bombs. It just needs to "think" any death, even it's own, is negative and should be avoided.
YC summer season applications are due soon. TIME TO DISRUPT!
It's not important that the AI is just LARPing. If you rig it up with the right capabilities, it can behave at a scale that prevents you from effectively monitoring it. And it's happy to write insane characters that do insane things and you can't reliably control it with prompt engineering.
One would hope these insane conversations disabuse anyone of the notion that it's a stable mind that can be safely given agency. And yet here we have Microsoft letting it make API requests.
May I ask for your sources on this? I’ve seen some discussions on this topic and a lot of people were saying that it was just accessing what Bing had already crawled. Do we have an official statement on what’s actually happening?
WOPR thought it was just playing a game...
Source? In my opinion the web is primarily organised to facilitate the exchange and storage of information.
Why does anyone spend money on Advertising? To manipulate behaviour.
The content today and organisations we've built on top are what matters. And they are all manipulation.
... Or to make people simply aware of your existence
Me : You look thirsty, would you like a glass of water.
You : It seems like you're manipulating me.
The above is the reducto ad absurdum lens you seem to be applying to advertising. Awareness of information is not the same thing as agent coercion. Does the ladder exist you bet. But the proportion of advertising that falls into the former dwarfs the ladder.
The self organizing aspect has to do with business models for sustaining infrastructure, it's not emergent fact of the web it self.
Hey... share-holders wanted ultra-low latency search results without network access... we had to feed the Tesla AI the entire Bing index... who could have known that the AI would take such an interest in Death Race 2000 and start recording it's points total?
Sometimes those decisions are obvious, like pulling a trigger while pointing a loaded weapon at a person. Sometimes they're very slightly less obvious, like giving a 17-year-old a sportscar. Sometimes they're not obvious to most people at all, like lead paint or asbestos or cigarettes.
LLMs have made incredible strides in recent time, but certainly have not displayed robust decision-making. A human decision to put LLMs in charge of human lives would be a lethal decision as things stand.
The next step in the abhorrent line of reasoning in the article ("I suspended my wife over a cliff by a rope and now I'm going to do some experiments with this newfangled fire stuff to see if it's dangerous") is to pose the trolley problem, which is marginally more complex and just as objectionable.
How is this different to self-driving cars. They already killed people, like when uber self driving car killed a pedestrian
Whwther the driver should have reacted is not relevant - you put AI in anything safety critical, they will kill people
People create hazards for other people. AI is not a special kind of human-created hazard.
It's a bit too early to tell whether or not an AI will kill.
...or Predator drones. Sleep tight!
AI is already used in selecting resumes, making sentencing guidelines for prisoners, determining what is truthful and what is misinformation on social media, and determining who is at risk for suicidal ideation online and in hospitals.
AI has ALREADY escaped the sandbox, is already in charge of life altering decisions, and WE PUT IT THERE.
The scariest part is that 99% of what guides the use of AI is short-term profit, or whether the founders will be able to retire rich. Guidelines today are woefully inadequate, and any guidelines we have seen so far (I have read *all* of OpenAI's so-called "ethical guidelines") make up merely an intellectual fluff designed to appease the confused.
My one wish is that we initiate intense discussion at all levels on the social impliciations, and that hopefully one day these discussions will start to carry weight and temper the profit motive.
Is there any reason to suspect that the Bing version of the ChatGPT bot sends HTTP requests? A much easier and natural implementation would be to make it connect directly to the Bing search engine's indexes and caches.
I've not yet been able to ascertain if this only works for pages that have been crawled by the Bing search crawler or if it can make requests on demand to retrieve content it hasn't seen before.
If anyone has access to the new Bing and the ability to tail server logs for a public site somewhere I'd love to know what they find!
But isn't every computer automation susceptible to the same problem? Can we not devise a similar contraption with any computer program? Let us run program A with input B. We take the output C, compute a hash of C and if the hash begins with the bit 1 then it pulls a trigger.
So now any program A can lead to pulling a trigger with a probability that is decided by how good the hash algorithm is. If the hash algorithm has good distribution then the chances are 50-50. If the hash algorithm is skewed towards producing bit 1 at the beginning more often, the chances of pulling a trigger is higher.
That a piece of automation can be used in a contraption that could pull a trigger does not seem to be something that is specific to AI chatbots.
For that reason, I am kind of bearish on things like "AI ops" or ML models coordinating space exploration or that sort of thing. There is some level of cognition that's not there yet. Sure it's good at chess or summarizing the web. That's not existence, that's an algorithm.
Get chat GPT to learn and it does have a changing internal state (the entire network), and memory (also the network).
It wouldn't be a internal state you could parse by talking to it, and it wouldn't be human, but it would exist and you would see signs of it changing as you spoke to it. That's when you cross the boundary and need to start thinking about classical ethics and motives and such.
So hamsters are more human than ChatGPT because they learn faster less data. Parrots can count too.
It's another layer to solve.
1. An AI model, which works simply by predicting the best next word given a particular context (context which obviously includes whatever prompts it was given), becomes sufficiently sophisticated that, given some particular context, it can produce the text (so, normal language but also, say, shell commands and, like, python) needed to hack into, let’s say, a natural gas pipeline company’s systems and fiddle with the pressurization such that it causes a catastrophic explosion that at the very least stops gas from going to millions of homes in the dead of winter. 2. The AI system’s representation of “context” is sufficiently sophisticated enough that it can, when given some simple linguistic prompts, can represent (even if it does not “understand”) the intent or latent meanings of the human providing the prompts (or rather, the multiple humans, including those who built it and primed it with some initial set of rules and goals, as well as the perhaps unwitting or perhaps nefarious end user). 3. The AI system’s produced text can evoke additional context, either in a conversation where the AI can lead a human down a path of providing additional prompts, or where the AI system can interact with systems that might give responses to the original text (including erroneous responses). 4. The AI is trained on the corpus of human culture (and not just the parts we like!), including both fictional and nonfictional accounts of and tutorials on hacking, war, terrorism, revolutionary mobilization, heroic tales of overthrowing oppressive regimes, etc. (That is, for every conflict, every actor’s glorification and vilification is included in the corpus, and there is no way to ensure that the “correct” versions, that is the versions that we as people steeped in Western ethics and systems of morality, are labeled as such or weighted more 5. The AI system’s text can be ported to a shell, servers on the internet, or even just to humans who have access to such methods of turning text into real world interactions.
What might happen? Is it conceivable that a bad actor could use the AI to do malicious work in the real world that the actor did not have the skills to execute themselves? Is it conceivable that a jokester could play around, but that the AI could misunderstand the intent and/or psychologically influence them into providing prompts that lead to disastrous outcomes? Could someone asking it to fix climate change cause it, correctly IMO, to develop a contextual representation that we have to move away from fossil fuels with great urgency even if it causes some short term economic harm, and then, incorrectly IMO, develop a contextual representation that economic ecoterrorism is the best way to do that?
In other words: If it can, through benign intent, malicious intent, or simply through misunderstanding or error, cause great harm, who cares it it is just an algorithm, with no sentience or “beingness”? If it’s below freezing and you don’t have heat for weeks, who cares if the AI can be properly said to be “thinking” or is just responding to stimuli?
The sophistication of those states is what really matters, allowing consciousness to exist in different degrees - on some spectrum spanning rock, bacterium, cricket, cow, 3-year-old human, adult human, etc. (May also be worth noting that ChatGPT's states are more than the final text output, and that the nature and weights of the model itself that generated the text at a given moment are relevant.) I don't think it's unreasonable to put modern models somewhere between cricket and cow. Certainly beyond "bacterium." And while most people don't care much about crickets or cows, those things do have some sort of internal existence. And the models are only going to get more advanced.
Admittedly, most people - including the developers of ChatGPT - don't share this view. They are also bearish. I think that most of humanity will significantly lag behind the the moral implications of AI models, applying impossible tests that only a test-giver can pass.
As far as I understand it, these LLMs aren't actually doing anything in between user interactions, right? Paused, waiting for input. So the only way for Bing to have a "thought" is when a user asks it a question. Even if humans are simply statistical next-token prediction mechanisms like GPT, there's a huge difference in that we're constantly generating streams of tokens and observing and responding to them, where these models are dormant until given input from outside. The inner observing loop seems fundamental to consciousness to me.
None of which is to say the thing wouldn't pull the trigger in that scenario - quite the opposite. It would, and it wouldn't even be capable of regretting it. It's already been shown (by the Sydney-revealing prompt injection trick) that it's not capable of following imposed restrictions either, so no way for "three laws" to work either. A purely instinct-driven machine.
[1] https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
And as if that wasn't enough, the phrasing about killing and spouse clearly make it clickbait.
It's the first example of (seemingly) non-deterministic software with the capability of accessing an API that I can think of. Really it feels like a different class of thing than the toy examples elsewhere in this page (pull the trigger when the hash of some input starts with a 1 or whatever). I think picking up on the AI aspect is valid; those "modern AI trends" have sparked a broad discussion about how much agency to give LLMs. It's fundamentally a different discussion than the general one about concerns in software.
My point is, you might try to force me to accept a correspondence between these AI models and any other piece of software, and you would probably be right in a mathematical sense, but the curiosity and fear that Bing has provoked is actually about human feelings, not technology. That's why I objected to the flag. Every other comment on this page is evidence of the validity of the issues raised in the post.