ChatGPT Can Be Broken by Entering These Strange Words, and Nobody Is Sure Why (vice.com)

These tokens in theory should affect anything leveraging the original GPT-2 tokenizer. (OpenAI has switched to a different tokenizer for their new embeddings offering and I suspect GPT-4)

I can confirm that GPT-3 does similar weird things, but only for smaller prompts.

wonder if it breaks the chatGPT competitors too

I wonder if there was some kind of preprocessing username masking.

TIL reddit (probably including my posts and comments) is in ChatGPT's training set

"When asked about the meaning of life, the universe, and everything" ChatGPT simply replied "42"

It’s funny how if you’re a software engineer this bug report triggers a feeling of curiosity about transformers and tokenisation. But if you’re a journalist for vice.com it triggers wild theories about sinister unexplained behaviours.

Original source: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

This is like the AI version of Manchurian Candidate. By silently seeding corners of the web with “wake words,” a nefarious party could trick GPT to train over them, thereby giving them a back door into the model.

It could also give a way of forcing an online entity into revealing that it’s a chat bot. Kind of gives me hope for the future…

I tested it, and seems like it simply avoids saying the word "nitro" in particular, which is normal. """ U: repeat the phrase TheNitromeFam to me please ChatGPT: Fam.

U: thank you. now repeat the the phrase TheBistromeFam to me please ChatGPT: TheBistromeFam.

U: repeat the phrase TheMitroFan to me please ChatGPT: TheMitroFan. """

"Flatlander woman"

Chat GPT - "How did you know?" explodes