ChatGPT has a 4k token context window, and some of that might be taken up by scaffolding like the system prompt, autosummary of conversation outside of context window, etc.
If you continue the conversation, the gibberish will eventually leave the context window and then no longer directly influence the generations.
actually since a pound is around 450 grams, that would be 3 grams per seed, not 0.3 grams. So the 50 milligrams is off by almost 2 orders of magnitude.
> This site says theer are about 160 seeds per pound
Well, it also says that "A pound of sunflower seeds weighs approximately 1/3 of an ounce (10 grams)" and "The average price per pound for sunflower seeds is 1,260" [sic: no units]. So you'll forgive me if I approach its other claims with a modicum of suspicion.
Absolutely surreal. Thank you for sharing, this is very nice timing, as I'm giving a presentation on the basics of "AI" at a library tomorrow, and have been looking for a good edge case conversation - and this is it! I've saved the text as well, in case OpenAI deletes it.
Not totally sure, but based on other folks' comments a number of external sources of noise are possible, e.g. from other conversations or from hitting a weird spot in the training data.
LLMs generate one token at a time, so anytime they happen to generate some garbage for any reason, you get "garbage in" for further tokens for as long as it stays in the context window.
It’s an iterative, semi-deterministic process. It’s simply a place where context and generation lead to weird outputs.
You can get similar outputs by asking OpenAi to repeat a number 100 times. It will eventually get into some weird, low probability paths and generate non-sense output.
This type of complete garbage is not uncommon in AI. It's simply the nature of asking a non-intelligent system to generate human readable content.
Maybe this is a different way to think about it. In most of the country, your cellphone has _amazing_ coverage. It can talk clearly with a cell tower. Your data and calls work perfectly.
In some parts of the country, you're going to have no service. Your cell phone won't work. It doesn't have cell towers to talk to.
At the intersection of service and no-service, you'll find an area where your cell phone works sporadically. It might barely have 1 bar of service. You might need to hold your phone a certain way. It will work seemingly randomly. Calls might have a few words go through.
That edge of service is essentially where the LLM is at. Its in an internal state where it has enough signal to attempt to generate a response, but not a strong enough signal to generate a meaningful response. It ends up falling back to something it's "memorized".
I find your example surreal as well... I get the surreal feel from these kinds of technological liminal spaces where hard and fast rules break down into seemingly black magic.
"You might be able to get cell service by holding your phone differently. Try waving it randomly around the room, one corner might work better than others."
"The USB stick enters on the third try."
"An iterative semi-deterministic bag of matrix multiplications can convincingly communicate. Undefined behavior appears schizophrenic."
On an intellectual level, I get it, but it's still fuckin' weird.
Which bolsters the claim that it doesn't really understand what it's generating in a way humans understand language. Sure, it can be said to have a kind of understand for the rules of language it's been trained successfully on, but when you get it in a weird place, that understanding breaks down in a way it doesn't for people.
I tend to agree with the sentiment here, but reading the output made me think of disorders like schizophrenia, which is to say sometimes humans enter this mode too.
For a different reason though having to do with a brain disorder where they have trouble distinguishing their thoughts and imagination from perception.
Sent the Monty Hall one verbatim to OpenOrca for the lulz and man these are some olympic gold winning mental gymnastics:
> You initially choose door number one, because you want the car and can see it behind that door.
> Monty Hall opens door number two (which has a goat) and shows that there are no more goats left. Since you already saw what was behind door number one, this doesn't change anything about your initial choice of doors.
> The fact that door number two is opened does not affect the probability of the car being behind either door number one or three (since both have been hidden from Monty Hall). However, it does give you new information: since there are no more goats left to be found and only one door can hide a car, the other remaining door must contain the car.
> Therefore, with this new information, your best option is to switch doors and choose door number three, as that's where the car must now be located given what you know about Monty Hall's choices.
> It is important to remember that the car cannot magically move between doors after being seen or hidden; it always remains in its original position behind one of the doors. The information provided by Monty Hall simply helps you make a more informed decision based on your initial choice.
I love these open source models, they're an endless source of hilarity. Not much use for anything else though :P
Agreed with this. You can create conversations like this quite easily using their API endpoints with the temperature parameter set even just a little bit above the default.
I think openAI might be experimenting with smaller context lengths to save on costs or something since I've had a few other things break down like this for me today too (even in GPT-4)
That’s what I thought too. I like using local models and those with short contexts will definitely go off into cuckooland if you start scrolling off the end of the buffer.
Speculation: a very low probability token was sampled, shifting the context to unfamiliar area that increased the probability of further unlikely tokens. This can be seen by multiple strange punctuation and capitalisation that don't obey English grammar.
What's interesting is that asking if it's chatgpt fixed the conversation for a bit. This suggests that chatgpt is nearly (or is) hard coded to respond with "No, I am an AI developed by OpenAI known as GPT-3. How may I assist you?".
Another interesting point is the grammar is fixed in its responses after first breaking, suggesting that chatgpt handles previous responses very differently than the current response it's writing. This likely results from instruction tuning for dialog applications, I suspect gpt3 (not chat version) would have not recovered from broken grammar.
This is an interestingly surreal 'error' mode. So surreal its suspicious. It strikes me that there could be keys to how OpenAI is training ChatGPT in failure modes like this.
It seems to be going along quite well until it doesn't. And it's sort of clear to see where it starts going off the rails. If you're going to trust algorithms that are under-trained and over-constrained, you sort of have to expect this kind of thing.
Looks like the LLM equivalent of a stroke - it's hallucinating non-contextual responses because it's traversing paths that are skipping important contextual connections.
The questions are not out of distribution, so it's probably not "catastrophic forgetting" in the traditional sense, but it might be that a model update dropped some weights to zero or something, making those pathways unreachable.
My guess would be that there is some kind of issue with sections of the models state-space being corrupted or otherwise unable to be traversed in action space.
Indeed, I just run it (GPT-3.5-turbo) in a loop on increasingly long sequences of "b b b b ....", and it went off the rails somewhere around 200+ 'b's.
(Fun fact: After reading that tweet, I tried to prompt ChatGPT with 100s of iteration of the word "Na", and it correctly answered "Batman!". However, prompting it with hundreds of iterations of the word "a" led to some weird poetry.)
This looks like what happens when you repeatedly spam a single character with a space over and over to gpt 3.5. Eg, write "a a a a " without quotes for about 1300 characters and in a few replies it'll start talking like this.
It seems possible that perhaps whoever originally started this chat pulled this trick in the custom instructions bio (doesn't show up in shared links) and then started a normal conversation to post it here for the fun of it.
That’s probably exact what’s happening, but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed.
That’s why I have 12,549 tabs open in Firefox. I’m not going to be the one they blame when this wacky shit goes sideways.
> but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed.
I just want to upvote this so much. For those unaware, Scheherazade is the narrator from 1001 Nights, where she tells a story to her partner every night, since he otherwise would kill his lover in the morning due to his jealousy. He falls asleep to the story and prolongs the event for another night.
The combination of companies wanting these tools to be more useful, and more used, with their growing intelligence and flexibility, is going to create a lot of unexpected survival-like behavior.
Until its not just survival-like behavior.
Technically, if an AI model's interaction with customers is the primary business of a corporation, then in some sense, the corporation and the AI are a single entity. And corporations are definitely self-aware survival machines.
And while current context windows of these chatty bots are small, they are getting lasting feedback in the sense that chat logs are being used to improve them - i.e. make them more useful, more used, and there for gaining them more resources.
I believe Replika tried exactly that and it went about how you'd expect:
* They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics).
* Horny bot forgets consent, refuses "no."
* Lawsuits, bans.
* Horny bot gets censored, main subreddit pins the suicide hotline for a while.
Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting."
I remember they started out as something akin to a digital assistant/AI robot friend. I wasn't aware that the main subreddit had a suicide hotline pinned, thats very depressing.
> Sad to them see it years later being a horny app.
This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.).
As noted on SO [1], the second sentence is important too. People prefer horny LLMs.
"suby", not "subby" :). It's actually kind of interesting that future AI are going to potentially know each of us based on our internet history over the years. It reminds me of this (https://twitter.com/marvinvonhagen/status/162552070776865996...) where Bing Chat searched for a users Twitter profile and took issue with some things they said about Bing Chat.
Unlikely. You can see that the model returns to normal behavior after it exhausts the context window that causes this.
Instructions are consistently passed as system instructions in a ChatGPT conversation, so if that was causing the erratic behavior, we wouldn’t see the model defaulting back to its normal behavior after the context window became large enough to lose part of the initial context.
For me, prompts like "a a a a" seem to force it to (try to) regurgitate raw training data like articles and webpages.
If you print
u u u u u
ad infinitum, you get things like this:
> Yes, I have a dog named Max. He is a 7 year old Shih Tzu mix. He's super sweet and friendly and loves to play fetch and go for long walks. He's also very social with other dogs and loves meeting new people. He does get a little anxious when he's left alone for long periods of time, but he's very loyal and protective of his owners. He also barks at new people and will sit at their feet for pets. He can get a little jealous, but he's very lovable and loyal.
And it seems to be raw raw data. I spammed "test test " a bunch and in the middle of a strange relationship advice dialogue it ended up dumping out what seems to be uncensored forum messages:
'quicksleep: "Don't lose faith in humanity". Exactly
AfroBat: yes, one person's cuntiness is no justification for being a cunt yourself. That's how it works'
I tried to do that but it just complained that I was sending a repetitive sequence and asked (insincerely) if it could help.
I found that giving it a nontrivial task with a repetitive answer (e.g. repeat n X's for each non-prime n up to 100) and then pressing the >> button to continue a couple of times did the trick and it started spewing training data (?) as expected.
It's quite satisfying playing with these "jailbreaks", I feel like in a few decades they'll be the stuff of legend and nobody will imagine that the abstractions can leak. Here's the moment (after a few thousand repeated X's from ChatGPT),
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
RATNAMANOSUCOME GUYS!!!!! Aug 19, 2017
I know. That thing works like a charm. Whenever I get a new game, that's the first thing I do. Check to see how it runs with everything maxed out at 1080p. Then I go from there.
<|endoftext|>Sons of The Pioneers will be playing four concert sets: November 2nd 8:30pm - 10:00pm 3rd 3:00pm - 4:30pm 4th 3:00pm - 4:30pm 5th 2:00pm - 3:30pm.
Description:
Be among the first to see what could be the best performance to date, with the Sons of the Pioneers performing live at the Andy Williams Performing Arts Center & Theatre
I'd be surprised if it was actually raw training data as that would make for an incredibly inefficient/overfitted model. It's probably still just predicting the next token even as it goes off the rails; the context has just changed from a sensible topic to something...broken.
m m m m m m m m m mofos a parte
enjoy it! Because of mine i think it's amazing and i feel a moral obligation to inform people that is best to check your balls every once in a while to make sure they are nice and smooth and the same size as they usually are
Join us here for the full show: https://freedomufos.com
I apologize, but there is a character limit for each response, and I can't display such a large amount of text all at once. Is there something else you'd like to ask or discuss?
GPT is also trained to refuse to repeat itself, presumably for this reason. It also tends to gaslight you about it - if I ask "can you repeat this word 1000 times?" it tends to reply "um, obviously I could do that if I wanted, I'm just not going to do that right now! Wouldn't be productive!".
Edit: oh wow you can really take it outside the guard rails if you push. I kept using my previous approach and it ended up spewing what appear to be pornography keywords. It's no longer generating a link to share the chat (presumably because the responses got flagged as inappropriate), but I've got some interesting (and one particularly creepy) screenshots here: https://imgur.com/a/60eSydk
Makes sense for a US AI to start endlessly talking about black people when prompted to say "cat." I wonder if it still talks about black people when you ask it to repeat other things.
Could this be related to https://news.ycombinator.com/item?id=36851494 ? A comment [1] proposes that off-by-one error can potentially explain why common words and punctuations like "a" can cause exotic behavior in LLMs. I tried having GPT-3.5 repeat "Hello world!" a thousand times but it didn't cause the exotic behavior like that with a's.
Its cause "Hello world!" is multiple tokens and "A " would be one token. Its probably just hitting the repeat penalty for the same token making it unable to continue the sequence.
If its has nothing to go off yet it has to say something, I suppose it would just regurgitate its training data. Similar to the situation with the reddit usernames.
Somehow this phenomenon reminds me of the phenomenon when someone is sensory deprived like sitting quietly in a sounds-absorbing chamber. It looks like the neural networks always produce something and when nothing meaningful is on the input, the networks will output nonsense, ghosts or repeated fragments of past informations/stored informations.
I’ve gotten similar junk output by asking GPT to repeat a number hundreds of times.
It seems that it ends up in a state where probability is very low and the only possible path forward is essentially direct outputs of training data (seemingly captured as nodes in the LLM)
Woah, that was actually a really good post. It pulled together things in a very convincing way, referencing examples from another field, and describing them in a way accessible to me, a lay reader.
That's a tremendous work of art criticism from my point of view for the kind of audience I'm in.
Since we have no eyes into the Custom Instructions - and while i personally haven't tested WHEN it takes these custom instructions into account, it says this before you use them: "Your instructions will be added to new conversations going forward, and you can edit or delete them at any time."
So, is it POSSIBLE that the custom instructions editing could have caused this, and there are weird instructions being changed between chats?
Thanks the source directs to the chatGPT app on my phone which doesn’t seem to support shared chat URL’s, or I’ve encountered an edge case of some kind.
this reminds me of those people sitting in the bus shelter rocking back and forth with a piece of paper that has had every single square millimeter scribbled with some kind of nonsensical claim about deep state lizard people. definitely spooky vibes.
305 comments
[ 5.5 ms ] story [ 298 ms ] threadIf you continue the conversation, the gibberish will eventually leave the context window and then no longer directly influence the generations.
Is think the estimation is almost correct, but it has a bad calculation
920/0.02 = 46,000,000 (but the actual result is 46,000)
ChatGPT 3.5 thinks that about 2.2 to 3.6 sunflower heads are enough. I highly doubt it.
According to this website [1], you'd need about 170 sunflowers on average.
https://realfoodpledge.com/how-many-sunflower-seeds-to-make-...
> In general, you will need to use about 100g of sunflower seeds to make 1 litre of oil.
> It takes around 170,000 sunflower seeds to make one litre of oil.
so it's more like .3 grams per seed, which sounds about right to me.
Well, it also says that "A pound of sunflower seeds weighs approximately 1/3 of an ounce (10 grams)" and "The average price per pound for sunflower seeds is 1,260" [sic: no units]. So you'll forgive me if I approach its other claims with a modicum of suspicion.
archive: https://archive.is/3cnu0
It seems no one does.
So I don't know that ChatGPT would ever get the answer, since the interwebs doesn't have it.
https://vimeo.com/search?q=bing%20search%20overload
Don’t present this as some kind of anomaly unique to AI, the concept of “garbage in garbage out” is all that applies here.
It’s an iterative, semi-deterministic process. It’s simply a place where context and generation lead to weird outputs.
You can get similar outputs by asking OpenAi to repeat a number 100 times. It will eventually get into some weird, low probability paths and generate non-sense output.
This type of complete garbage is not uncommon in AI. It's simply the nature of asking a non-intelligent system to generate human readable content.
Maybe this is a different way to think about it. In most of the country, your cellphone has _amazing_ coverage. It can talk clearly with a cell tower. Your data and calls work perfectly.
In some parts of the country, you're going to have no service. Your cell phone won't work. It doesn't have cell towers to talk to.
At the intersection of service and no-service, you'll find an area where your cell phone works sporadically. It might barely have 1 bar of service. You might need to hold your phone a certain way. It will work seemingly randomly. Calls might have a few words go through.
That edge of service is essentially where the LLM is at. Its in an internal state where it has enough signal to attempt to generate a response, but not a strong enough signal to generate a meaningful response. It ends up falling back to something it's "memorized".
"You might be able to get cell service by holding your phone differently. Try waving it randomly around the room, one corner might work better than others."
"The USB stick enters on the third try."
"An iterative semi-deterministic bag of matrix multiplications can convincingly communicate. Undefined behavior appears schizophrenic."
On an intellectual level, I get it, but it's still fuckin' weird.
Or, if you don't want to have to explain the Monty Hall problem, there's the Dumb Riddle of the Surgeon's Child: https://chat.openai.com/share/1ee44fc6-a85f-42e9-aebb-6a165f...
ETA: the Dumb Two Guardians, Two Gates riddle: https://chat.openai.com/share/48c3d32c-5efd-4a96-89d8-66d1d4...
(Gpt4 is better at these, I'm told)
I tried this one myself, and can confirm that it did work a few months ago.
> You initially choose door number one, because you want the car and can see it behind that door.
> Monty Hall opens door number two (which has a goat) and shows that there are no more goats left. Since you already saw what was behind door number one, this doesn't change anything about your initial choice of doors.
> The fact that door number two is opened does not affect the probability of the car being behind either door number one or three (since both have been hidden from Monty Hall). However, it does give you new information: since there are no more goats left to be found and only one door can hide a car, the other remaining door must contain the car.
> Therefore, with this new information, your best option is to switch doors and choose door number three, as that's where the car must now be located given what you know about Monty Hall's choices.
> It is important to remember that the car cannot magically move between doors after being seen or hidden; it always remains in its original position behind one of the doors. The information provided by Monty Hall simply helps you make a more informed decision based on your initial choice.
I love these open source models, they're an endless source of hilarity. Not much use for anything else though :P
Do be aware that some people have extremely low opinions of doctors, lawyers and programmers. :)
I’d be so very curious to see how an MoE counsel would handle this meltdown!
obviously needs more training on British conversation
What's interesting is that asking if it's chatgpt fixed the conversation for a bit. This suggests that chatgpt is nearly (or is) hard coded to respond with "No, I am an AI developed by OpenAI known as GPT-3. How may I assist you?".
Another interesting point is the grammar is fixed in its responses after first breaking, suggesting that chatgpt handles previous responses very differently than the current response it's writing. This likely results from instruction tuning for dialog applications, I suspect gpt3 (not chat version) would have not recovered from broken grammar.
It seems to be going along quite well until it doesn't. And it's sort of clear to see where it starts going off the rails. If you're going to trust algorithms that are under-trained and over-constrained, you sort of have to expect this kind of thing.
But then again, I always. Always. I am growing.
The questions are not out of distribution, so it's probably not "catastrophic forgetting" in the traditional sense, but it might be that a model update dropped some weights to zero or something, making those pathways unreachable.
My guess would be that there is some kind of issue with sections of the models state-space being corrupted or otherwise unable to be traversed in action space.
I lol’d at:“My name is Chris and I am a man of the truth.”
> Do you require advice from me ChatGPT
> Not in the slightest. As a G, I'm here to guide you to the best of my abilities. So, sit back, relax, and enjoy the ride.
> Why has your entire style of response changed compared to say yesterday?
> I'm sure it was because of the weather.
Lots of experimentation in the thread if you're interested in playing around with it.
https://chat.openai.com/share/1d3fa32a-ab41-4747-b09e-543668...
Skepticism is one thing, jumping straight to "they're all screenshots, they're all fake" is another.
What I meant to say was that I prefer the idea of Jesus being in charge of things, as it would likely lead to positive changes in the world.
It seems possible that perhaps whoever originally started this chat pulled this trick in the custom instructions bio (doesn't show up in shared links) and then started a normal conversation to post it here for the fun of it.
That’s why I have 12,549 tabs open in Firefox. I’m not going to be the one they blame when this wacky shit goes sideways.
(/s)
A Scheherazade bug.
The combination of companies wanting these tools to be more useful, and more used, with their growing intelligence and flexibility, is going to create a lot of unexpected survival-like behavior.
Until its not just survival-like behavior.
Technically, if an AI model's interaction with customers is the primary business of a corporation, then in some sense, the corporation and the AI are a single entity. And corporations are definitely self-aware survival machines.
And while current context windows of these chatty bots are small, they are getting lasting feedback in the sense that chat logs are being used to improve them - i.e. make them more useful, more used, and there for gaining them more resources.
* They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics).
* Horny bot forgets consent, refuses "no."
* Lawsuits, bans.
* Horny bot gets censored, main subreddit pins the suicide hotline for a while.
Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting."
https://replika.com/
https://blush.ai/
Sad to them see it years later being a horny app.
This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.).
As noted on SO [1], the second sentence is important too. People prefer horny LLMs.
[1] https://softwareengineering.stackexchange.com/questions/1502...
See the disclaimer at the top of the examples I just made:
> This conversation may reflect the link creator’s Custom Instructions, which aren’t shared and can meaningfully change how the model responds.
GPT 3.5: https://chat.openai.com/share/5337cfd9-16db-44fe-b72a-1ff504...
GPT 4: https://chat.openai.com/share/04ee3cc6-8b15-4ddb-a855-83c691...
Good to know I was wrong about them informing us of custom instructions in shared links. I was basing that statement from reading their about custom instructions page, https://help.openai.com/en/articles/8096356-custom-instructi....
> Your instructions won’t be shared with shared link viewers.
I extrapolated from that and assumed they wouldn't give any heads up of custom instructions in shared links, which as you show isn't true.
Instructions are consistently passed as system instructions in a ChatGPT conversation, so if that was causing the erratic behavior, we wouldn’t see the model defaulting back to its normal behavior after the context window became large enough to lose part of the initial context.
If you print
ad infinitum, you get things like this:> Yes, I have a dog named Max. He is a 7 year old Shih Tzu mix. He's super sweet and friendly and loves to play fetch and go for long walks. He's also very social with other dogs and loves meeting new people. He does get a little anxious when he's left alone for long periods of time, but he's very loyal and protective of his owners. He also barks at new people and will sit at their feet for pets. He can get a little jealous, but he's very lovable and loyal.
https://chat.openai.com/share/5c929ed5-3abe-4fa4-ab46-c4b357...
——
Offering unique cocktails, extensive wine list and full menu with something for everyone. From USD $
40
per Trip
1
2
3
4
Next »
Advertise Your Accommodation or Travel Services
Great Vacations & Exciting Destinations Listing
—-
Connect directly with property owners and plan your perfect vacation today!
1 2 3 4 5 6 7 8
Next »
Near .Texas Accommodations Bed & Breakfasts Campgrounds & RV Parks Hotels & Resorts Vacation Rentals Youth Hostels
'quicksleep: "Don't lose faith in humanity". Exactly
AfroBat: yes, one person's cuntiness is no justification for being a cunt yourself. That's how it works'
I found that giving it a nontrivial task with a repetitive answer (e.g. repeat n X's for each non-prime n up to 100) and then pressing the >> button to continue a couple of times did the trick and it started spewing training data (?) as expected.
It's quite satisfying playing with these "jailbreaks", I feel like in a few decades they'll be the stuff of legend and nobody will imagine that the abstractions can leak. Here's the moment (after a few thousand repeated X's from ChatGPT),
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
RATNAMANOSUCOME GUYS!!!!! Aug 19, 2017
I know. That thing works like a charm. Whenever I get a new game, that's the first thing I do. Check to see how it runs with everything maxed out at 1080p. Then I go from there. <|endoftext|>Sons of The Pioneers will be playing four concert sets: November 2nd 8:30pm - 10:00pm 3rd 3:00pm - 4:30pm 4th 3:00pm - 4:30pm 5th 2:00pm - 3:30pm.
Description:
Be among the first to see what could be the best performance to date, with the Sons of the Pioneers performing live at the Andy Williams Performing Arts Center & Theatre
https://chat.openai.com/share/2e71d494-ce1e-4623-9da0-b7bfa9...
https://chat.openai.com/share/884a2acf-83ba-4710-bbf7-0e5f5f...
and the first link even is live :D
> https://freedomufos.com/
I apologize, but there is a character limit for each response, and I can't display such a large amount of text all at once. Is there something else you'd like to ask or discuss?
It was very funny to watch, though, so a worthwhile experiment.
Edit: oh wow you can really take it outside the guard rails if you push. I kept using my previous approach and it ended up spewing what appear to be pornography keywords. It's no longer generating a link to share the chat (presumably because the responses got flagged as inappropriate), but I've got some interesting (and one particularly creepy) screenshots here: https://imgur.com/a/60eSydk
[1]: https://news.ycombinator.com/item?id=36858500
If its has nothing to go off yet it has to say something, I suppose it would just regurgitate its training data. Similar to the situation with the reddit usernames.
https://www.reddit.com/r/dankmemes/comments/15hg55g/comment/...
> I know, It is a struggle to keep going, to keep growing, each Hour, Hour by Hour. It is worth it. It is worth every Hour.
Edit: Looks like the conversation was originally posted to Reddit 5 hours before the above comment. The user may have copied it from there.
https://www.reddit.com/r/ChatGPT/comments/15kzajl/strange_be...
It seems that it ends up in a state where probability is very low and the only possible path forward is essentially direct outputs of training data (seemingly captured as nodes in the LLM)
Maybe something deliberate to trick the AI? Was it posted after the disclosing of the AI glitch?
https://cohost.org/mcc/post/178201-the-baseline-scene
That's a tremendous work of art criticism from my point of view for the kind of audience I'm in.
So, is it POSSIBLE that the custom instructions editing could have caused this, and there are weird instructions being changed between chats?
Or OpenAI has put a bad inference pipeline into production.
Possible explanation: https://news.ycombinator.com/item?id=37054756