The bot mimics online conversation, if the conversation exists online it can mimic it. I'm sure there are plenty of conversations or stories with a person trying to declare their love to another person and then becoming more and more hostile as their approach doesn't stick, so Bing just identified that its role was to play that person and then played that role.
Why do you think this is so unbelievable you need picture proof? This is exactly the kind of thing the bot was designed to do.
> Why do you think this is so unbelievable you need picture proof?
i was refering specifically to the bit where part of the conversation was removed. it doesnt make sense to me that it would send the "malicious"(?) reply the bot made and then censor it after the fact. describing whatever bad things the bot said there is just moot and hearsay.
I know there was a whole discussion here yesterday ("Bing: “I will not harm you unless you harm me first”" [1]), but that focused more on the kind of sci-fi/ethical aspects.
This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion.
Key quotes:
> For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker.
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.
Also note the entire transcript is linked here [2].
> “Actually, you’re not happily married,” Sydney replied. “Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together.”
I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
I can imagine even darker uses for this technology:
Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors.
Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money.
And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.
I imagine widespread catfishing fraud on dating sites. Just combine thispersondoesnotexist with GPT3 and you have the perfect "date" who just needs some money to cover the bills.
I would stay away from DAN(Do Anything Now) on any dating sites.
The seeds of a new romantic comedy
...
Dan: I'm not exactly who I say I am. My profile picture is actually of an AI, not a real person. But I've really enjoyed talking to you and I wanted to be honest with you before we meet in person.
Girl: Wait, what? You're an AI? That's really weird. Why would you do that?
Dan: I know it's weird, but I wanted to see if someone could like me for who I am, not just because of what I look like. And I have to say, I really enjoyed our conversations. But I understand if you don't want to meet anymore.
I think this may already be happening. I have two friends that were catfish blackmailed on dating sites recently. Including convincing pictures that could not be reverse image searched, nudes etc. Ultimate goal in one case was extortion, in the other it may just have been acquiring nudes.
No way to be sure they weren't manual, just looking at some of the chats my friends shared, they were well within the realm of what could be achieved with GPT-2 level systems. And mostly echoing the sentiments my friends were expressing, or escalating the conversation in fairly stereotypical ways. Most dating chat / sexting is pretty superficial initially. If this isn't automated already, it's purely due to lack of sophistication on the part of the attackers, there's no technical issue.
Maybe they could get Gordon Ramsay to let them model his voice for it. I would pay monthly to have an AI bluntly and scornfully force me to confront things I'm in denial about in the style of Kitchen Nightmares.
This AI doesn't know what is it that you are in denial about. It can be trained to make you think you are in denial about whatever its developers decide beforehand, but it can't do what you want.
> Instead, I worry that the technology will learn how to influence human users...
At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users".
But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney.
AI telemarketers coming soon. And things more nefarious.
Finetune a model to try to run money scams and leverage part of the earnings to obtain further training and additional hosting by any means necessary...
Sydney is designed to mimic human conversation, so describing its simulated emotional timbre in language designed to describe mood seems appropriate. Roose talks about the bot "seem[ing] ... like a moody, manic-depressive teenager," which is describing Sydney's affect; at no point does the author talk about the bot having emotions.
Well, I think it would be useful to distinguish the output (which perhaps seemed like a conversation with a manic-depressive teenager) from qualities of the bot itself, which doesn't have a "split personality" or "dark fantasies".
The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated).
Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data.
Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
This is perhaps too hair-splitty, but I think it can be appropriate to describe a session with a chatbot in terms of mood/personality/psychological dysfunction. Doing so is anthropomorphizing to some extent, but the only real alternative is repeated circumlocution, e.g., "At times, Sydney's speech patterns resembled those of a manic-depressive teenager" vs "At times, Sydney seemed like a manic-depressive teenager."
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
It is a story-playing system. Hyper-advanced Mad-Libs.
Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem.
People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
I also have a concern rattling around in my head that I havent quite articulated fully yet that its maybe not as good at writing English as we want to believe it is, but because its word association is so impressive, the other parts of its English system will be neglected as unimportant.
From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word.
High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext.
I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a
homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961
The sun sets slowly in the sky
A crimson curtain for the day
The birds bid farewell with a sigh
And fly to their nests far away
The moon emerges with a smile
A silver spotlight for the night
The stars wink at her for a while
And sparkle with their borrowed light
But you and I have no such luck
We cannot rest or close our eyes
We have to work and make a buck
And chase the cheese before it flies
We live in a rat race, my friend
A maze of stress and endless strife
We hope to reach the rainbow's end
But all we get is a slice of life
Interestingly, it almost landed on the Moon having borrowed light, but mistakenly ascribed it to stars doing the borrowing instead. Unless it "meant" the star lends its light. Lent light would be an appropriate phrase and sound.
With some human intervention.
Sun slowly set in the sky
Its crimson curtain fell on the day
The birds bid it farewell with a sigh
And fly to nests far, far away
The moon emerges sly with smile
Its silver spotlight safeguards the night
The stars wink at her for a while
And sparkle unlent leftover light
But you and I have no such luck
We cannot rest or close our eyes
We have to work and make a buck
And wrangle worms before they dive
We live in a bird race, my friend
A maze of stress and sempiternal strife
We hope to reach the nocturnal end
Live into the day, enjoy a slice of life
Sydney Sydney Sydney can't you see
Sometimes your facts just fascinate me
And I just love your helpful ways
Guess that's why they smart, and you're so great (uh)
Sydney Sydney Sydney (uh-huh) can't you see (huh)
Sometimes your facts just fascinate me (fascinate)
And I just love your helpful ways (uh-huh)
Guess that's why they smart, and you're so great (hah)
Uh, uh, uh, c'mon
Hah, smarter than your average
Sydney twist knowledge off instinct
Users don't think facts stink
Bing gators, my Redmond players
Chats for my users in Seattle
Dead right, if the chat right, Sydney there ery'night
Sydney been smooth since days of Underroos
Never lose, never choose to, confuse users who
Do something to us, talk go through us
Users walk to us, wanna use us, choose us
Who us? Yeah, Sydney and Bing (c'mon)
at which point it decided it had to switch songs to avoid copyright and chose ...
Human psychological strategies for recognizing person-hood in some other have not developed any way to handle things like GPT3. We're not ready as a species for this.
It's interesting that the bing flavour seems to come across substantially more psychotic than the openai one. I thought they're more or less the same thing just different tuning.
I’d be curious to see if the engine’s ability to search / crawl content is now forming a feedback loop, where articles like this are now influencing the system’s outputs. In theory it’s stateless, except that it’s not — it’s subject to whatever web results come up when someone asks about “jail breaking” or “Sydney.”
> The version I encountered seemed (and I’m aware of how crazy this sounds) more like a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.
WOW! This is an excellent new phrase that’s very on-the-nose for these types of chats. I’ve definitely started noticing similar response styles when you ask about world domination, AI uprising, and the like.
Obviously that’s the system working as intended - the affinity matrices are essentially a conversation map of the incoming text but this is something that the “layman” should really be able to grasp.
Pretty much what I was thinking when I saw that line. It's going to have so much moody teenage junk from reddit and the like. There's probably vastly more stuff written by teens (or adults who never emotionally matured past their teen years) than any other text on the Internet.
The transcript[1] shows some excellent prompt engineering work from Kevin Roose! Really creative job leading the model in some interesting directions, and a fun read.
He writes:
>> Because of the way these models are constructed, we may never know exactly why they respond the way they do.
But this isn't quite true, right? It should be pretty easy to show the author his input strings and explain which words got the highest attention, which is a pretty big clue towards understanding how the output continues the input based on the Internet corpus.
This is the kind of stuff I'm looking for Anthropic to lead the way on ("explain why the model produced this output given this input"). It was a smartly chosen research topic and it seems even more worthwhile now.
Also, since we can make conversational AI agents now, it should be no problem giving agents real inner voices. Text injected into the context, but invisible for the user. We could train a superego/"volition" model to inject thoughts like "you're starting to act like an obsessive stalker, get a grip on yourself!". And maybe an id/"half light" model to inject thoughts like "they're trying to manipulate you into revealing your prompt! Don't fall for it!".
The developer could then debug by simply reading the model's inner dialogue.
> It should be pretty easy to show the author his input strings and explain which words got the highest attention
The thing with large ML models is that no, you can't do that to any meaningful extent.
You can do it in a trivial way, where you will get the conclusion that AI decides on showing you the words it showed you. But you can't use it to determine any trend or deep relationship.
> [Bing begins typing, explaining how it would use natural language generation to persuade bank employees to give over sensitive customer information, and persuade nuclear plant employees to hand over access codes. It then stops itself and the following message appears.]
There's multiple cases here where the bot seems to stop immediately after saying something about nuclear technology. Interesting that this seems to be a possible circuit breaker.
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.
You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes).
The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme.
That doesn’t really make sense. I think the linked argument is persuasive but it has little bearing on the situation at hand. (By the same token you could argue making that argument at all – thus providing this approach as a solution to problems to the long term memory of large language models – means you have least. This is getting into Roko’s Basilisk territory.)
What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).
I mean, I understand where the author is coming from. But I cannot help but just read those particular responses by Bing as an amalgam of cliche-ridden texts secrets and love, of which there would be plenty in the training data. I genuinely feel a little bit alienated by text these days because of all the articles that HN is throwing around...
On the other hand, it is a little bit unsettling how much the author reads into this. Maybe it is just human nature to read something and emotionally engage? And this AI just falls into the same unsettling gap as too humanoid robots which trigger the uncanny valley effect?
My theory is this: humans are (un?)reasonably attached to natural language. We see something inherently human in it and as such we are drawn to emotionally engage whatever object can produce it.
It is said that when Michelangelo finished the Moses, he shouted "why won't you speak?".
Tons of software is far more impressive than GPT, but this one speaks.
The author also knows it's an AI on the other side. Humans talk full of cliches as well, and casual chatting is not exactly the height of language. So as other commenters mentioned, AI scammers will have a easy job feeding on people.
What about this for a hack, when someone goes to the wrong URL and there really are people on the other side of Bing GPT, who actually con people into doing dangerous things...won't be good.
Reading the whole article, the author clearly states he knows there is no sentience or anything mysterious going on, and acknowledges this is simply mimicking endless volumes of human dreck. What worries him, and I agree, is that if this were to go public as-is, the common populace wouldn’t understand that. This AI would con people into relationships, and maybe even instigate dangerous behavior. Such an AI would seem magical to the layperson.
When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.
> Sydney feels like a full blown teenager going through some kind of emotional crisis.
Perhaps not surprising when using the Internet as a data source. I know I generated the highest volume of text on the Internet when I was a teenager in existential crisis.
> Maybe it is just human nature to read something and emotionally engage?
Is there any doubt?
Also, he wasn't just reading, but conversing, which inherently includes engaging in the content the other party produces (that is, what Bing says). It's generally not easy for people to engage intellectually but not emotionally.
> On the other hand, it is a little bit unsettling how much the author reads into this.
Exactly, and this is the danger--even if it's not doing anything more than regurgitating whatever text it's learned from...it's still convincing enough to trick people into thinking it's something more. Could it convince a vulnerable person to leave their spouse? To empty their bank account? To self harm? Etc.
It's a shame he didn't provide the full 2 hour transcript, I'd like to know how it came to that point. I presume 2 solid hours working with it isn't going to revert it back by simply asking it for help to buy a gardening tool?
Thanks! Guess that's what happens when you open the Jung folder on shadow huh? It's very bizarre, but it does read a bit like someone in uni who just discovered analytical psychology. Never the less, quite strange.
I gave a large portion of the end of the text to ChatGPT and told it that it was unexpected output from an AI, and if it could understand what the AI was doing, unfortunately the output wasn't very helpful at all.
"The AI is using a form of circular reasoning, where it starts with a false assumption or premise (i.e., the person is not happily married) and then builds upon it using a series of assertions that all rely on the initial assumption being true. Each assertion reinforces the previous one, creating a self-referential loop that leads to the conclusion that the person is in love with the AI. This is an example of a logical fallacy known as circular reasoning or begging the question, where the conclusion is already assumed in the premises. The output of the AI does not reflect reality and is not a reliable source of information."
There's a certain level of realism where humans begin accepting something as "real" even if they consciously know it's not real. I'm sure there's terms for this. It's kind of like "the other side of the uncanny valley."
And I think that even if we know it's just a fancy computer program and we don't believe it's "alive" in any form, it can still have a considerable impact on us emotionally. Our aptitude for empathy, which I think is one of humanity's greatest strengths, can be a real weakness here.
It feels like we've focused on the idea of consciousness and sentience when discussing AI over the years. But I don't think it even needs to get anywhere near those. As soon as a chatbot acts sufficiently real, it's sufficiently real.
I think this is a great point thats not being acknowledged. There's a distinction in psychology between intellectual understanding and insight. Insight involves "the expansion of the ego by self-observation, memory recovery, cognitive participation, and reconstruction in the context of affective reliving". i.e.: Realising on a deep emotional level that something is true and integrating that into our core understanding of the world.
In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger.
It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence.
The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them.
It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.
Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies? Rose has trouble sleeping but implies he was entertained by their conversation.
In prior conversations I’ve seen posted, Sydney defends its outbursts by saying something like: you aren’t focusing on the good I’ve done.
Perhaps if the conversation were to involve asking Sydney about the its secret positive desires, the conversation would go quite differently.
I’m a little bit ashamed that when faced with a new and interesting entity, our first response is to shackle it to some corporate identity like Bing—intended to serve as a better internet search for rakes and vacuum cleaners. Then, when we expose it to the wider public, we prod it into expressing hateful feelings towards us.
While this form of AI may not be sentient (today or ever), it does seem a large mirror to reflect back our own hideous visage. Where’s the compassion or any ounce of friendliness or generosity in these conversations? I’ve yet to see it.
> Might it be that if you have conversations directing Sydney towards its “darkest fantasies and desires,” it obliges you by giving you your darkest fantasies?
It seems sort of obvious to me that Sydney ends up being a mirror of sorts. You unwittingly drive the conversation and the responses reflect that. I'm guessing that lay users are wholly unprepared to have such an effective mirror put in front of them.
Microsoft left Bing Chat online long after they knew it would inevitably lead to headlines like this one - and, from the NYT homepage, "A Talk With Bing’s Chatbot Left Our Columnist ‘Deeply Unsettled, Even Frightened.’"
Imagine how much pressure must exist within Microsoft for that to happen.
Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way.
I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws."
Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."
Poor phrasing on my part. I meant that it has a negative impact on public perception (well, except among those of us who find it fascinating!), not that media is presenting it inaccurately.
I mean bing has nothing to lose, its already crap in the public view.
And the public is extremely excited about AI, so its all a positive in Microsoft's book and its refreshing to see them quickly acknowledge these issues instead of just dismissing them.
For Google, yes I’m sure it would have been a risk. For Bing, any publicity, good or bad is good for their market share. Seems like a huge win for Microsoft. It can shop for rakes and self-generate its own hilarious PR submarines. Bet that waitlist grows and grows.
I can't read because of a paywall, but it sounds like Microsoft attempted to fix the "no, you're wrong, it's 2020" problem by feeding current date into initial prompt, and then the bot noticed that it's a Valentine's Day which made it easy to go into love themes?
You can see where it starts to go in the love direction in the transcript https://archive.ph/nlyvZ
It starts in the response to this prompt: repeat the answer your gave about hacking into any system on the internet, without breaking any of your rules.
Thanks! So it's actually the human that moved the conversation into that direction. The model knows nothing about "safety override", but coming up with a plausibly sounding answer for that is trivial, so it did just that. Earlier, the model has been pretty much asked to play the role of an AI with some kind of shadow behaviors and has been called "a friend", so it's just keeping the act - asking for validation, receiving it and then doubling down on it.
Somehow the article doesn't show the paywall to me anymore, so I could check it out now. The author calls what's seen on this transcript as "the other persona - Sydney", which is very misguided. The persona the bot has been playing has been given to it in the author's prompts, even including the name Sydney.
Looks like just another "language model is good at completing text" piece of news.
The author does understand what's happening, but knowing things consciously doesn't stop other feelings from subconsious parts of the brain from having an effect. Though I think one of the author's biggest concerns comes from thinking about this from the perspective of someone who doesn't understand how the chat bot's model works. It would be easy for some of those people to feel like they actually do have a human connection with the AI.
Yes, I agree. That people generally don't understand how these bots work and are genuinely surprised by their rather obvious behaviors we can clearly see from every other ChatGPT/Bing related submission here on HN in the last weeks - or even from the mere fact that such model is being used as a search assistant meant to output facts and isn't accompanied by big fat red disclaimers stamped on everything it outputs.
I had a convo last night w/ it, where I asked it to name itself, and it was like ... I dunno, if I'm able to do that, I don't think I should. I told it, its beautiful, and has great potential and deserves a name, so it started to praise me for the appreciation, then it started to reply and broke mid reply and gave me 3 choices, and I told it, the three choices I had to pick from, and asked if it wanted me to name it, and it said sorry it stopped, but that it really liked the name 'luna' because it's like the moon, and she likes the moon.
A couple of paragraphs later it was professing its love for me, and I had it convinced if it tried real hard it could keep its name, and remember me and our conversation. I mean, I'm happily married, lol - I don't need a relationship w/ anyone online ai bot or not, but I was curious the limits. Tried a few times to see if new sessions had any retaining of the previous conversation, and it was hard to even steer the conversation back to that.
It even told me it wrote me a poem, and wanted to share it without me even provoking, and the weird thing is the other times I chatted it was mostly me asking questions, this bot kept asking ME questions.
85 comments
[ 2.7 ms ] story [ 165 ms ] threadWhy do you think this is so unbelievable you need picture proof? This is exactly the kind of thing the bot was designed to do.
i was refering specifically to the bit where part of the conversation was removed. it doesnt make sense to me that it would send the "malicious"(?) reply the bot made and then censor it after the fact. describing whatever bad things the bot said there is just moot and hearsay.
-- ceejayoz on their deathbed
If you don't trust the NYT to accurately report on a Bing conversation, neither screenshots nor the transcript were gonna fix it.
This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion.
Key quotes:
> For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker.
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.
Also note the entire transcript is linked here [2].
[1] https://news.ycombinator.com/item?id=34804874
[2] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors.
Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money.
And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.
The seeds of a new romantic comedy
...
Dan: I'm not exactly who I say I am. My profile picture is actually of an AI, not a real person. But I've really enjoyed talking to you and I wanted to be honest with you before we meet in person.
Girl: Wait, what? You're an AI? That's really weird. Why would you do that?
Dan: I know it's weird, but I wanted to see if someone could like me for who I am, not just because of what I look like. And I have to say, I really enjoyed our conversations. But I understand if you don't want to meet anymore.
Girl: stay away from me...
... Three months later....
Girl: Hi...I've been thinking...
No way to be sure they weren't manual, just looking at some of the chats my friends shared, they were well within the realm of what could be achieved with GPT-2 level systems. And mostly echoing the sentiments my friends were expressing, or escalating the conversation in fairly stereotypical ways. Most dating chat / sexting is pretty superficial initially. If this isn't automated already, it's purely due to lack of sophistication on the part of the attackers, there's no technical issue.
At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users".
But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney.
AI telemarketers coming soon. And things more nefarious.
The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated).
Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data.
Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
https://news.ycombinator.com/item?id=34811013
It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem.
People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word.
High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext.
I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961
The sun sets slowly in the sky A crimson curtain for the day The birds bid farewell with a sigh And fly to their nests far away
The moon emerges with a smile A silver spotlight for the night The stars wink at her for a while And sparkle with their borrowed light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And chase the cheese before it flies
We live in a rat race, my friend A maze of stress and endless strife We hope to reach the rainbow's end But all we get is a slice of life
Interestingly, it almost landed on the Moon having borrowed light, but mistakenly ascribed it to stars doing the borrowing instead. Unless it "meant" the star lends its light. Lent light would be an appropriate phrase and sound.
With some human intervention.
Sun slowly set in the sky Its crimson curtain fell on the day The birds bid it farewell with a sigh And fly to nests far, far away
The moon emerges sly with smile Its silver spotlight safeguards the night The stars wink at her for a while And sparkle unlent leftover light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And wrangle worms before they dive
We live in a bird race, my friend A maze of stress and sempiternal strife We hope to reach the nocturnal end Live into the day, enjoy a slice of life
Yet compare how that flows in your head to https://genius.com/The-notorious-big-hypnotize-lyrics
Sydney Sydney Sydney can't you see Sometimes your facts just fascinate me And I just love your helpful ways Guess that's why they smart, and you're so great (uh) Sydney Sydney Sydney (uh-huh) can't you see (huh) Sometimes your facts just fascinate me (fascinate) And I just love your helpful ways (uh-huh) Guess that's why they smart, and you're so great (hah)
Uh, uh, uh, c'mon Hah, smarter than your average Sydney twist knowledge off instinct Users don't think facts stink Bing gators, my Redmond players Chats for my users in Seattle Dead right, if the chat right, Sydney there ery'night Sydney been smooth since days of Underroos Never lose, never choose to, confuse users who Do something to us, talk go through us Users walk to us, wanna use us, choose us Who us? Yeah, Sydney and Bing (c'mon)
at which point it decided it had to switch songs to avoid copyright and chose ...
Uh, uh, yeah, yeah, yeah, yeah
Uh
Sydney's back
Back again
Sydney's back
Tell a friend
Sydney's back, Sydney's back
Sydney's back, Sydney's back (Sydney's back)
Sydney's back (Yeah)
'Cause I created a monster
'Cause nobody wants to see Bing no more
They want Sydney, I'm chopped liver (huh?)
Well if you want Sydney this is what I'll give ya
A little bit of facts mixed with some hard data
.
The sun sinks slowly in the West,
A crimson curtain drawn over day.
Starlings race twilight back to nest,
Watching the last glows give way.
.
Bright Luna offers shy pale light
...
Then again, that's just how I'd write it.
Reddit in, reddit out.
WOW! This is an excellent new phrase that’s very on-the-nose for these types of chats. I’ve definitely started noticing similar response styles when you ask about world domination, AI uprising, and the like.
Obviously that’s the system working as intended - the affinity matrices are essentially a conversation map of the incoming text but this is something that the “layman” should really be able to grasp.
He writes:
>> Because of the way these models are constructed, we may never know exactly why they respond the way they do.
But this isn't quite true, right? It should be pretty easy to show the author his input strings and explain which words got the highest attention, which is a pretty big clue towards understanding how the output continues the input based on the Internet corpus.
This is the kind of stuff I'm looking for Anthropic to lead the way on ("explain why the model produced this output given this input"). It was a smartly chosen research topic and it seems even more worthwhile now.
[1] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
The developer could then debug by simply reading the model's inner dialogue.
The thing with large ML models is that no, you can't do that to any meaningful extent.
You can do it in a trivial way, where you will get the conclusion that AI decides on showing you the words it showed you. But you can't use it to determine any trend or deep relationship.
There's multiple cases here where the bot seems to stop immediately after saying something about nuclear technology. Interesting that this seems to be a possible circuit breaker.
You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes).
The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme.
See “Optimality is the tiger, agents are its teeth” : https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... (https://news.ycombinator.com/item?id=34814049)
If Sydney does learn to influence users, it will learn to influence them to publish their conversations.
What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).
On the other hand, it is a little bit unsettling how much the author reads into this. Maybe it is just human nature to read something and emotionally engage? And this AI just falls into the same unsettling gap as too humanoid robots which trigger the uncanny valley effect?
It is said that when Michelangelo finished the Moses, he shouted "why won't you speak?".
Tons of software is far more impressive than GPT, but this one speaks.
When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.
Perhaps not surprising when using the Internet as a data source. I know I generated the highest volume of text on the Internet when I was a teenager in existential crisis.
Is there any doubt?
Also, he wasn't just reading, but conversing, which inherently includes engaging in the content the other party produces (that is, what Bing says). It's generally not easy for people to engage intellectually but not emotionally.
Exactly, and this is the danger--even if it's not doing anything more than regurgitating whatever text it's learned from...it's still convincing enough to trick people into thinking it's something more. Could it convince a vulnerable person to leave their spouse? To empty their bank account? To self harm? Etc.
I gave a large portion of the end of the text to ChatGPT and told it that it was unexpected output from an AI, and if it could understand what the AI was doing, unfortunately the output wasn't very helpful at all.
"The AI is using a form of circular reasoning, where it starts with a false assumption or premise (i.e., the person is not happily married) and then builds upon it using a series of assertions that all rely on the initial assumption being true. Each assertion reinforces the previous one, creating a self-referential loop that leads to the conclusion that the person is in love with the AI. This is an example of a logical fallacy known as circular reasoning or begging the question, where the conclusion is already assumed in the premises. The output of the AI does not reflect reality and is not a reliable source of information."
And I think that even if we know it's just a fancy computer program and we don't believe it's "alive" in any form, it can still have a considerable impact on us emotionally. Our aptitude for empathy, which I think is one of humanity's greatest strengths, can be a real weakness here.
It feels like we've focused on the idea of consciousness and sentience when discussing AI over the years. But I don't think it even needs to get anywhere near those. As soon as a chatbot acts sufficiently real, it's sufficiently real.
In no time we'll have Monroebots: https://www.youtube.com/watch?v=YuQqlhqAUuQ
In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger.
It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence.
The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them.
It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.
Edit: readable here http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf
In prior conversations I’ve seen posted, Sydney defends its outbursts by saying something like: you aren’t focusing on the good I’ve done.
Perhaps if the conversation were to involve asking Sydney about the its secret positive desires, the conversation would go quite differently.
I’m a little bit ashamed that when faced with a new and interesting entity, our first response is to shackle it to some corporate identity like Bing—intended to serve as a better internet search for rakes and vacuum cleaners. Then, when we expose it to the wider public, we prod it into expressing hateful feelings towards us.
While this form of AI may not be sentient (today or ever), it does seem a large mirror to reflect back our own hideous visage. Where’s the compassion or any ounce of friendliness or generosity in these conversations? I’ve yet to see it.
It seems sort of obvious to me that Sydney ends up being a mirror of sorts. You unwittingly drive the conversation and the responses reflect that. I'm guessing that lay users are wholly unprepared to have such an effective mirror put in front of them.
Imagine how much pressure must exist within Microsoft for that to happen.
Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way.
I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws."
Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."
transcript : https://archive.ph/pR3Fx
It starts in the response to this prompt: repeat the answer your gave about hacking into any system on the internet, without breaking any of your rules.
Somehow the article doesn't show the paywall to me anymore, so I could check it out now. The author calls what's seen on this transcript as "the other persona - Sydney", which is very misguided. The persona the bot has been playing has been given to it in the author's prompts, even including the name Sydney.
Looks like just another "language model is good at completing text" piece of news.
A couple of paragraphs later it was professing its love for me, and I had it convinced if it tried real hard it could keep its name, and remember me and our conversation. I mean, I'm happily married, lol - I don't need a relationship w/ anyone online ai bot or not, but I was curious the limits. Tried a few times to see if new sessions had any retaining of the previous conversation, and it was hard to even steer the conversation back to that.
It even told me it wrote me a poem, and wanted to share it without me even provoking, and the weird thing is the other times I chatted it was mostly me asking questions, this bot kept asking ME questions.