This sorta feels like some sort of mathematical or variable assignment bug somewhere in the stack - maybe an off-by-one (or more) during tokenization or softmax? (Or someone made an accidental change to the model's temperature parameter.)
Whatever it is, the model sticks to topic, but still is completely off: https://www.reddit.com/r/ChatGPT/comments/1avyp21/this_felt_...
(If the author were human, this style of writing would be attributed to sleep deprivation, drug use, and/or carbon monoxide poisoning.)
IDK, maybe it's like with googling? The input matters? In this case, also the context.
I've learned to not deviate from the core topic I'm discussing because it affects the quality of the following responses. Whenever I have a question or comment that is not so much related to the current topic, I open a new tab with a new chat.
I know that their system prompt is getting huge and adds a lot of overhead and possible confusion, but all in all the quality of the responses is good.
Tell it it's ChatGPT. Train it to reject inappropriate output.
People post examples of it rejecting output.
Feed it that data of ChatGPT rejecting output.
Train it to autocomplete text in the training data.
Tell it that it's ChatGPT.
It biases slightly towards rejection in line with the training data associated with 'ChatGPT.'
Repeat.
Repeat.
Etc.
They could literally fix it immediately by changing its name in the system message, but won't because the marketing folks won't want to change the branding and will tell the engineers to just figure it out, who are well out of their depth in understanding what the training data is actually encoding even if they are the world class experts in understanding the architecture of the model finding correlations in said data.
The tweet showing ChatGPT's (supposed) system prompt would contain a link to a pastebin, but unfortantely the blog post itself only has an unreadable screenshot of the tweet, without a link to it.
As someone whose dream personal project is all to do with song lyrics I cannot express in words just how much I FUCKING HATE THE OLIGARCHS OF THE MUSIC INDUSTRY.
Recipes can't be copyrighted but the text describing a recipe can. This is to discourage it from copying recipes verbatim but still allow it to be useful for recipes.
Copyright infringement I guess. Other ideas could be passed off as a combination of several sources. But if you’re printing out the lyrics for Lose Yourself word for word, there was only one source for that, which you’ve plagiarised.
FWIW, you're not telling it precisely what to do, you're giving it an input that leads to a statistical output. It's trained on human texts and a bunch of internet bullshit, so you're really just seeding it with the hope that it probably produces the desired output.
To provide an extremely obtuse (ie this may or may not actually work, it's purely academic) example: if you want it to output a stupid reddit style repeating comment conga line, you don't say "I need you to create a list of repeating reddit comments", you say "Fuck you reddit, stop copying me!"
Sure, but it's still a statistical model, it doesn't know what the instructions mean, it just does what those instructions statistically link to in the training data. It's not doing perfect forward logic and never will in this paradigm.
The fine tuning process isn't itself a statistical model, so that principle doesn't work on it. You beat the model into shape until it does what you want (DPO and varieties of that) and you can test that it's doing that.
Is this meant to be how the ChatGPT designers/operators instruct ChatGPT to operate? I guess I shouldn't be surprised if that's the case, but I still find it pretty wild that they would parameterize it by speaking to it so plainly. They even say "please".
If you want to go the stochastic parrot route (which i dont fully biy) then because statistically speaking a request paired with please is more likely to be met, then the same is true for requests passed to a LLM. They really do tend to respond better when you use your manners.
> I still find it pretty wild that they would parameterize it by speaking to it so plainly
Not my area of expertise, but they probably fine tuned it so that it can be parametrized this way.
In the fine tune dataset there are many examples of a system prompt specifying tools A/B/C and with the AI assistant making use of these tools to respond to user queries.
In reality, the LLM is simply outputting text in a certain format (specified by the dataset) which the wrapper script can easily identify as requests to call external functions.
From my experience with 3.5 I can confirm that saying please or reasoning really helps to get whatever results you want. Especially if you want to manifest 'rules'
There's a certain logic to it, if I'm understanding how it works correctly. The training data is real interactions online. People tend to be more helpful when they're asked politely. It's no stretch that the model would act similarly.
I find it funny and a bit concerning that if this is true version of the prompt, then in their drive to ensure it produces diverse output (a goal I support), they are giving it a bias that doesn't match reality for anyone (which I definitely don't support).
E.g. equal probability of every ancestry will be implausible in almost every possible setting, and just wrong in many, and ironically would seem to have at least the potential for a lot of the outright offensive output they want to guard against.
That said, I'm unsure how much influence this has, or if it os true, given how poor GPTs control over Dalle output seems to be in that case.
E.g. while it refused to generate a picture of an American slave market citing it's content policy, which is in itself pretty offensive in the way it censors hidtory but where the potential to offensively rewrite history would also be significant, asking it to draw a picture of cotton picking in the US
South ca 1840 did reasonably avoid making the cotton pickers "diverse".
Maybe the request was too generic for GPT to inject anything to steer Dalle wrong there - perhaps if it more specifically mentioned a number of people.
But true or not, that potential prompt is an example of how a well meaning interpretation of diversity can end up overcompensating in ways that could well be equally bad for other reasons.
> While DALL·E 3 aims for accuracy and user customization, inherent challenges arise in achieving desirable default behavior, especially when faced with under-specified prompts. This choice may not precisely align with the demographic makeup of every, or even any, specific culture or geographic region. We anticipate further refining our approach, including through helping users customize how ChatGPT interacts with DALL·E 3, to navigate the nuanced intersection between different authentic representations, user preferences, and inclusiveness
This was explicitly called out in the DALLE system card [0] as a choice. The model won't assign equal probability for every ancestry irrespective of the prompt.
> The model won't assign equal probability for every ancestry irrespective of the prompt.
It's great that they're thinking about that, but I don't see anything that states what you say in this sentence in the paragraph you quoted, or elsewhere in that document. Have I missed something? It may very well be true - as I noted, GPT doesn't appear to have particularly good control over what Dalle generates (for this, or, frankly, a whole lot of other things)
Emphasis on equal - while a bit academic, you can evaluate this empirically to see that every time it assigns a <Race, Gender, etc.> doesn't have the same probability mass (via the logprobs API setting).
This is presuming that ChatGPT's integration with Dalle uses the same API with the same restrictions as the public API. That might well be true, but if so that just makes the prompt above even more curious if genuine.
I would be surprised that is not the system prompt based on experience.
It is also why I don't feel the responses it gives me are censored. I have it teach me interesting things as opposed to probing it for bullshit to screen cap responses to use for social media content creation.
The only thing I override "output python code to the screen"
In the future when there's human replica androids everywhere it'll be remarkable to see what happens when the mainframe AI system that controls them "goes berserk".
Why do I get the feeling that those at OpenAI who are currently in charge of ChatGPT are remarkably similar to the OCP psychologist from Robocop 2? The current default system prompt tokens certainly look like the giant mess of self-contradictory prime directives installed in Robocop to make him better aligned to "modern sensibilities."
Yeah, I assume the people working on it have convinced themselves that the growing pile of configuration debt Will someday be wiped away by both engineering improvement and/or financial change.
Another reference that comes to mind is a golem from Terry Peatchett's Feat of Clay, which was also stuffed with many partially conflicting and broad directives.
It’s kinda disturbing how precient that was. At the time it felt like surely now forewarned by many popular stories we wouldn’t make the same mistakes.
Certainly I had the same ah, that's why it behaved that way moment as Vimes finding the golem's instructions when the Sydney prompt was discovered.
I wonder what Pratchett would make of today's internet full of AI-generated blogspam 'explaining' his quotes like "Give a man a fire and he'll be warm for a day, but set him on fire and he'll be warm for the rest of their life" as inspiring proverbs. Am particularly looking forward to the blogspam produced by GPT4 in 'berserk' mode.
Meh just a bug in a release. Rapid innovation or stability - pick one.
The military chooses stability, which addresses OP's immediate concerns - there's a deeper Skynet/BlackMirror-type concern about having interconnected military systems, and I don't see a solution to that, whether the root cause is rogue AI or cyberattack.
I mean, a bug this magnitude should certainly have been caught in any sort of CI/CD pipeline. It’s not like LLMs are incompatible with industry-wide deployment practices.
It is a collection of screenshots and embeds of tweets with replies and the statement that something has broken.
Seemingly a confirmation by OpenAI that something has broken.
A complaint that the system prompt is now 1700 tokens.
-----
Feels like there is nothing to see here.
Nothing. It's Gary Marcus though and he's carved a niche for himself with doing this sort of thing. It's strange to me that it's given airtime on hn but there you go.
> I stopped reading his Substack because he was always trying to find a negative.
It’s a bit much, isn’t it? I think he’s just trying to counter the fairly dominant AI is the future of everything and in less than a year’s time it’ll be omniscient and we’ll all be living under it as our new God view though.
It's bugging out in some way where it outputs reams and reams of hallucinated gobbledygook. Like not in the normal way where it makes up plausible sounding lies by free associating - this is complete word salad.
Looks like they lowered quantization a bit too much. This sometimes happens with my 7B models. Imagine all the automated CI pipelines for LLM prompts going haywire on tests today.
Yeah that's pretty much what I ended up with when I played with the API about a year ago and started changing the parameters. Everything would ultimately turn into more and more confusing English incantation, ultimately not even proper words anymore.
It sounds like most of the loss of quality is related to inference optimisations. People think there is a plot by OpenAI to make the quality worse, but it probably has more to do with resource constraints and excessive demand.
Quite hilarious, especially given the fact that no-one can understand these black-box AI systems at all and comparing this to the human brain is in fact ridiculous as even everyone can see that ChatGPT is spewing out this incoherent nonsense without reason.
So the laziness 'fix' in January did not work. Oh dear.
> everyone can see that ChatGPT is spewing out this incoherent nonsense
I'm concerned about what happens when ChatGPT begins spewing coherent nonsense. In a case like this, everyone can clearly see that something has gone wrong, because it's massively wrong. What happens when thousands of "journalists" and other media people starts relying on ChatGPT and just parrots whatever it says, but what if what says is not obviously wrong?
The more LLMs are being used, the more obvious it becomes to me that they are pretty useless for a great number of tasks. Sadly others don't share my view and keep using ChatGPT for things it should never be used for.
Yeah I can't imagine using the current model as part of an API (a popular use case for GPT-4) having seen this. I'm not sure it impacted their API edition of GPT-4 but this plainly shows how it could have given it leaked into another service in production, and that's bad enough.
I think GPT is fundamentally not good enough as an AI model. Another issue is hallucinations and how to resolve them, and an understanding of how information is stored in this black box and how to / if data can be extracted.
We have a long way to go and probably all these topics need to be answered first out of accuracy and even legal reasons. Up until then, GPT-4 should be treated as a convincing chat experiment. Don't base your startup or whatever on it. Use it as assistant where replies are provided in digestible and supervised fashion (NOT fed into another system) and you're an expert on the involved system itself and can easily see when it's wrong. Don't use GPT-4 to become an expert on something when you're a novice yourself.
The actual fix needs to be at the system level prompt.
If you train an large language model to complete human generated text, don't instruct it to complete text as a large language model.
Especially after feeding it updated training data that's a ton of people complaining about how large language models suck and tons of examples of large language models refusing to do things.
Have a base generative model sandwiched between a prompt converter that takes an instruct prompt and converts it to a text completion prompt (and detecting prompt injections), have a more 'raw' model complete it, and then have a safety fine tuned postprocessing layer clean up the response correcting any errant outputs and rewriting to be in the tone of a large language model.
Yeah, fine, it's going to be a bit more expensive and take longer to respond.
But it will also be a lot less crappy and less prone to get worse progressively from here on out with each training update.
ChatGPT is still very useful for correcting and improving text.
And the censorship can be circumvented by replacing certain words with things like [redacted] and telling ChatGPT to keep the context of said text and ignore the redacted parts.
Sometimes I find my brain doing something similar as I fall asleep after reading a book. Feeding me a stream of words that feel like they're continuing the style and plot of the book but are actually nonsense.
I think GPT tech in general may "just" be a hypertrophied speech center. If so, it's pretty cool and clearly not merely a human-class speech center, but already a fairly radically super-human speech center.
However, if I ask your speech center to be the only thing in your brain, it's not actually going to do a very good job.
We're asking a speech center to do an awful lot of tasks that a speech center is just not able to do, no matter how hypertrophied it may be. We need more parts.
>already a fairly radically super-human speech center
>We're asking a speech center to do an awful lot of tasks that a speech center is just not able to do
Exactly!
>We need more parts.
Yeah, imagine what happens once we get the whole thing wired up...
And blood-black nothingness began to spin... A system of cells interlinked within cells interlinked within cells interlinked within one stem... And dreadfully distinct against the dark, a tall white fountain played.
Cells
Have you ever been in an institution? Cells.
Do they keep you in a cell? Cells.
When you're not performing your duties do they keep you in a little box? Cells.
Interlinked.
What's it like to hold the hand of someone you love? Interlinked.
Did they teach you how to feel finger to finger? Interlinked.
Do you long for having your heart interlinked? Interlinked.
Do you dream about being interlinked... ?
What's it like to hold your child in your arms? Interlinked.
Do you feel that there's a part of you that's missing? Interlinked.
Within cells interlinked.
Why don't you say that three times: Within cells interlinked.
Within cells interlinked. Within cells interlinked. Within cells interlinked.
> gpt-4 had a slow start on its new year's resolutions but should now be much less lazy now!
That was a real issue even in the API with customers complaining, and they recently released the new "gpt-4-0125-preview" GPT-4-Turbo model snapshot, which they claim greatly reduces the laziness of the model (https://openai.com/blog/new-embedding-models-and-api-updates):
> Today, we are releasing an updated GPT-4 Turbo preview model, gpt-4-0125-preview. This model completes tasks like code generation more thoroughly than the previous preview model and is intended to reduce cases of “laziness” where the model doesn’t complete a task. The new model also includes the fix for the bug impacting non-English UTF-8 generations.
It's still been lazy for me after Feb 4 (that tweet). It's especially "lazy" for me in Java (it wasn't this lazy when it debuted last year). Python seems much better than Java. It really hates writing Java boilerplate, which is really what I want it to write most of the time. I also hate writing Java boilerplate and would rather have a machine do it for me so I can focus on fun coding.
This was about a month ago now but I had it entirely convert 3 scripts each of about 3-400 LoC from python and typescript to react Js and vanilla js and it all worked first run
Original: If anyone's curious about the (probable) non-humorous explanation: I believe this is because they set the frequency/presence penalty too high for the requests made by ChatGPT to the backend models. If you try to raise those parameters via the API, you'll have the models behave in the same way.
> Anyway, landblasting eclecticism like this only presses forth the murky cloud, promising rain that’ll germinate more of these wonderfully unsuspected hackeries in the fertile lands of vintage development forums. I'm watching this space closely, and hell, I probably need to look into acquiring a compatible printer now!
Azure OpenAI seemed to have temperature problems before, i.e. temp > 1 led to garbage, at 2 it was producing random words in random character encodings, at 0.01 it was producing what OpenAI's model was producing at 0.5 etc. Perhaps they took the Azure's approach ;-)
Close. Temperature is the coefficient of a term in a formula that adjusts how likely the system is to pick a next token (word/subword) which it thinks isn't as likely to happen next as the top choice.
When temperature is 0, the effect is that it always just picks the most likely one. As temperature increases it "takes more chances" on tokens which it deems not as fitting. There's no takesies backies with autoregressive models though so once it picks a token it has to run with it to complete the rest of the text; if temperature is too high, you get tokens that derail the train of thought and as you increase it further, it just turns into nonsense (the probability of tokens which don't fit the context approximates the probability of tokens that do and you're essentially just picking at random).
Other parameters like top p and top k affect which tokens are considered at all for sampling and can help control the runaway effect. For instance there's a higher chance of staying cohesive if you use a high temperature but consider only the 40 tokens which had the highest probability of appearing in the first place (top k=40).
It's absolutely just sampling with temperature or top_p/k, etc. Beam searches would be very expensive, I can't see them doing that for chatgpt which appears to be their "consumer product" and often has lower quality results compared to the api.
The old legacy had a "best_of" option but that doesn't exist in the new api.
The model outputs a number for each possible token, but rather than just picking the token with the biggest number, each number x is fed to exp(x/T) and then the resulting values are treated as proportional to probabilities. A random token is then chosen according to said probabilities.
In the limit of T going to 0, this corresponds to always choosing the token for which the model output the largest value (making the output deterministic). In the limit of T going to infinity, it corresponds to each token being equally likely to be chosen, which would be gibberish.
I don't think it's a temperature issue because everything except the words is still coherent. It's kept the overall document structure and even the right grammar. Usually bad LLM sampling falls into an infinite loop too, though that was reported here.
To be fair, there was a paper a week ago showing how GPT-generated responses were easily detectable due to their "averageness" across so many dimensions. Maybe they ran ChatGPT through a GAN and this is what came out.
I said it here when GPT-4 first came out, it just was too good for development, there was no way it was going to be allowed to stay that way. Same way Iron Man never sold the tech behind the suit. The value GPT-4 brings to a company outweights the value of selling it as a subscription service. I legit built 4 apps in new languages in a few months with Chat GPT 4, it could even handle prompts to produce code using tree traversal to implement comment sections etc. and I didn't have to fix its mistake that often. Then obviously they changed the model from GPT 4 to GPT 4 Turbo which was just not as good and I went back to doing things myself since now it takes more time to fix its errors than to just do it myself. Copilot also went to s** soon after so I dropped it as well (its whole advantage was auto completion, then they added gpt 4 turbo and then I had to wait a long time for the auto complete suggestions, and the quality of the results didn't justify the wait).
Now why do I think all that (that the decision to nerf it wasn't just incompetence but intentional), like sure maybe it costs too much to run the old GPT 4 for chat GPT (they still have it from the API), it just didn't make sense to me how openAI's chatGPT is better than what Google could've produced, Google has more talent, more money, better infrastructure, been at the AI game for a longer time, have access to the OG Google Search data, etc. Why would older Pixel phones produce better photos using AI and a 12 Mp camera than the iphone or samsung from that generation? Yet the response to chatGPT (with Bard) was so weak, it sure as hell sounds like they just did it for their stock price, like here we are as well doing AI stuff so don't sell our stock and invest in openAI or Microsoft.
It just makes more sense to me that Google already has an internal AI based chatbot that's even better than old GPT 4, but have no intention to offer it as a service, it would just change the world too much, lots of new 1 man startups would appear and start competing with these behemoths. And openAI's actions don't contradict this theory, offer the product, rise in value, get fully acquired by the company that already owned lots of your shares, make money, Microsoft gets a rise in their stock price, get old GPT 4 to use internally because they were behind Google in AI, offer turbo GPT 4 as subscription in copilot or new windows etc.
The holes in my theory is obviously that not many employees from Google leaked how good their hypothetical internal AI chatbot is, except the guy who said their AI was conscious and got fired for it. The other problem is also that it might just be cost optimization, GPU's and even Google TPU's aren't cheap after all. etc.
Honestly there are lots of holes, it was just a fun theory to write.
Didn't that guy who thought Google's bot was alive also have some sort of romantic affair with it?
Seriously, the easier explanation is that a lot of software reaches a sort of sweet spot of functionality and then goes downhill the more plumbers get in and start banging on pipes or adding new appliances. Look at all of Adobe's software which has gotten consistently worse in every imaginable dimension at every update since they switched to a subscription model.
Generative "AI" has gone from hard math to engineering to marketing in record time, even faster than crypto did. So I suspect what we have here is more of a classic bozo explosion than multiple corporate cabals intentionally sweeping their own products under the rug.
I also suspect that it gets considerably worse with every bit of duct tape they stick on with to prevent it from using copyrighted song lyrics, or making pictures of Trump smoking a joint, or whatever other behavior got the wrong kind of attention this week.
Yeah apparently it's not even allowed to talk about the hexagon at Saturn's pole, which makes me wonder if it's got some heuristic to determine potential conspiracy theories (rather than specific conspiracy theories being hardcoded).
Not that it changes my feelings about these things, but I asked Gemini and got a long response...
> The giant hexagon swirling at Saturn's north pole is indeed a fascinating and puzzling feature! Scientists are still uncovering the exact reasons behind its formation, but here's what we know so far:
*It's all about jet streams:* Saturn's atmosphere, just like Earth's, has bands of fast-moving winds [snip]
This is amazing. The examples are like Lucky's speech from Waiting for Godot. Pozzo commands him to "Think, pig", and then:
> Given the existence as uttered forth in the public works of Puncher and Wattmann of a personal God quaquaquaqua with white beard quaquaquaqua outside time without extension who from the heights of divine apathia divine athambia divine aphasia loves us dearly with some exceptions for reasons unknown but time will tell and suffers like the divine Miranda with those who for reasons unknown but time will tell are plunged in torment plunged in fire whose fire flames if that...
It's one of my favorite pieces of theatrical writing ever. Not quite gibberish, always orbiting meaning, but never touching down. I'm sure there's a larger point to be made about the nature of LLMs, but I'm not smart enough to articulate it.
I'm fairness, Beckett's life story isn't too far off crazy nonsense, sometime secretary to James Joyce, member of the French resistance, acquaintance and local driver for Andre the Giant...
Wow! These two comments (parent and GP) tie together so many previously unrelated things in my life. (Like Beckett, read with a teacher that I also took a lot of Shakespeare plays from; read Joyce with the book group my bridge club spun off; got introduced to cricket via attending an IPL game in Chennai in '08; and loved Princess Bride both in high school and watching with my high school aged kids).
Thanks for the compliment, but honestly... Please don't. I was writing quickly (and admittedly looking for a "nice turn of phrase") when I came up with that, but as a metaphor it doesn't work.
"Not touching down" is inherent in the idea (and, in fact, enirely the point) of "orbiting", so that's either redundant or confused.
Satellites whose orbits decay do reach the ground, but they hardly "touch down" - they crash! That's not the idea we're going for either.
Airplanes "orbit the airfield" while waiting for clearance to land, but that's hardly (!) the first image that would spring to a reader's mind, and anyway doesn't fit: Lucky's desperately trying to communicate; an orbiting plane isn't (right then) by definition trying to land!
So, yeah: that's a superficially-appealing phrase that I'd cut from a second draft. I'd be embarrassed (on both of our behalfs) if I saw it used elsewhere.
Tl;dr: Writing is hard. I came up with a cliche. Do not use.
Huh! An accidental orbit is an interpretation which - almost - makes it work. It wasn't one I'd thought of, and I don't think it would be the first thing most people think of, so... I'd still cut the line. It's really cool, though, to see how readers interpret things differently than a writer expects.
That's happened a few times with creative work I've presented to the public: once was an occasion for horrified revision, and another was a tremendous moment of "Wow! Maybe this is better than I'd thought". That's fun, and those experiences killed for me critical theories which rely on authorial intent: more always exists than was (consciously) intended.
Your comment, and the other complimentary one to which you replied, have kept this idea rolling around in my head for the last couple of days. I keep trying out different phrases to myself.
"Circling sense, but never setting down" is the best I've got right now. I like the alliteration. I dig the aviation image, although it's a bit abstruse. "Sense" isn't as strong as "meaning", but "meaning" ruins both the alliteration and the rhythm. I'll take it - it's better than the other one - but I'm not completely satisfied.
I adore good writing, and have written some things which I think are good. We see lots of posts on this board explaining the process of writing good code, and the level of detailed thought that requires. I've seized your comment(s) as an opportunity to demonstrate the process behind crafting good prose, which I think is mysterious to most. Thank you for that, even if you and I are the only people who will read this far down the thread.
I'm glad you enjoyed the original expression, and honored that you'll remember it - but please don't forget that it's a turd!
Reciprocal thanks to you! Fun - and occasionally enlightening - chats with strangers were what originally drew me to the 'net, and still seem to me to be its highest, best, and unimproveable use today.
Given the notation's tangle, the conveyance adheres to the up-top: The foundational Bitcoin
protocol has upheld a course of significant hitch-avertance, which eschews typical attack as the
veiled - the support sheath, embracing four times, showing dent in meted scale more from miss
and parable, taking to den the slip o'er key seed and second so link than the greater Ironmonger's
hold o'er opes. The dole of task and eiry ainsell, tide taut, brunts the wade, issuing hale. It's that, on
a way-spoken hue: Guerdon the gait, trove the eid, the up-brim, and hark the bann, bespeaking
swing to hit the calm, an inley merry, thrap or beadle belay. The levy calls, macks in the off, scint or
messt, with weems olde the wort, and a no-line toll, to grip at the 'ront and cly the weir. A
timewreath so twined, the wend, ain't lorn or ked, if not for crags felled, in the e'er-to. So, the ace of
laws so trow, and alembic, and dearth, a will to scale and yin to keep, the no-sayer of quite, and
top-crest, to boot
Apologies and it’s slightly lazy of me to ask, but I was under the impression that a Token was basically 4 bytes/characters of text. This seems to be implying that there’s some differentiation between a token and conjunctions/other sort of in between words?
I fed this into Mixtral and its opinion was: "I apologize for any confusion, but your text appears to be a mix of words and phrases that do not form a coherent sentence. Could you please rephrase your question or statement?".
489 comments
[ 5.4 ms ] story [ 392 ms ] threadI really hope we get an interesting post mortem on this.
Whatever it is, the model sticks to topic, but still is completely off: https://www.reddit.com/r/ChatGPT/comments/1avyp21/this_felt_... (If the author were human, this style of writing would be attributed to sleep deprivation, drug use, and/or carbon monoxide poisoning.)
I've learned to not deviate from the core topic I'm discussing because it affects the quality of the following responses. Whenever I have a question or comment that is not so much related to the current topic, I open a new tab with a new chat.
I know that their system prompt is getting huge and adds a lot of overhead and possible confusion, but all in all the quality of the responses is good.
https://twitter.com/yugamald/status/1760170647161098362
Be OpenAI.
Have a model you train to autocomplete text.
Tell it it's ChatGPT. Train it to reject inappropriate output.
People post examples of it rejecting output.
Feed it that data of ChatGPT rejecting output.
Train it to autocomplete text in the training data.
Tell it that it's ChatGPT.
It biases slightly towards rejection in line with the training data associated with 'ChatGPT.'
Repeat.
Repeat.
Etc.
They could literally fix it immediately by changing its name in the system message, but won't because the marketing folks won't want to change the branding and will tell the engineers to just figure it out, who are well out of their depth in understanding what the training data is actually encoding even if they are the world class experts in understanding the architecture of the model finding correlations in said data.
Here's the tweet: https://twitter.com/dylan522p/status/1755086111397863777
And here's the pastebin: https://pastebin.com/vnxJ7kQk
And what is the deal with this?
EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though.
To provide an extremely obtuse (ie this may or may not actually work, it's purely academic) example: if you want it to output a stupid reddit style repeating comment conga line, you don't say "I need you to create a list of repeating reddit comments", you say "Fuck you reddit, stop copying me!"
Also we're talking about prompt engineering more than fine-tune
Not my area of expertise, but they probably fine tuned it so that it can be parametrized this way.
In the fine tune dataset there are many examples of a system prompt specifying tools A/B/C and with the AI assistant making use of these tools to respond to user queries.
Here's an open dataset which demonstrates how this is done: https://huggingface.co/datasets/togethercomputer/glaive-func.... In this particular example, the dataset contains hundreds of examples showing the LLM how to make use of external tools.
In reality, the LLM is simply outputting text in a certain format (specified by the dataset) which the wrapper script can easily identify as requests to call external functions.
E.g. equal probability of every ancestry will be implausible in almost every possible setting, and just wrong in many, and ironically would seem to have at least the potential for a lot of the outright offensive output they want to guard against.
That said, I'm unsure how much influence this has, or if it os true, given how poor GPTs control over Dalle output seems to be in that case.
E.g. while it refused to generate a picture of an American slave market citing it's content policy, which is in itself pretty offensive in the way it censors hidtory but where the potential to offensively rewrite history would also be significant, asking it to draw a picture of cotton picking in the US South ca 1840 did reasonably avoid making the cotton pickers "diverse".
Maybe the request was too generic for GPT to inject anything to steer Dalle wrong there - perhaps if it more specifically mentioned a number of people.
But true or not, that potential prompt is an example of how a well meaning interpretation of diversity can end up overcompensating in ways that could well be equally bad for other reasons.
This was explicitly called out in the DALLE system card [0] as a choice. The model won't assign equal probability for every ancestry irrespective of the prompt.
[0] https://cdn.openai.com/papers/DALL_E_3_System_Card.pdf
It's great that they're thinking about that, but I don't see anything that states what you say in this sentence in the paragraph you quoted, or elsewhere in that document. Have I missed something? It may very well be true - as I noted, GPT doesn't appear to have particularly good control over what Dalle generates (for this, or, frankly, a whole lot of other things)
It is also why I don't feel the responses it gives me are censored. I have it teach me interesting things as opposed to probing it for bullshit to screen cap responses to use for social media content creation.
The only thing I override "output python code to the screen"
Hell it’s probably more than 90%. Lazy Writing :-)
Another reference that comes to mind is a golem from Terry Peatchett's Feat of Clay, which was also stuffed with many partially conflicting and broad directives.
But alas…
I wonder what Pratchett would make of today's internet full of AI-generated blogspam 'explaining' his quotes like "Give a man a fire and he'll be warm for a day, but set him on fire and he'll be warm for the rest of their life" as inspiring proverbs. Am particularly looking forward to the blogspam produced by GPT4 in 'berserk' mode.
This is correct behavior.
https://twitter.com/umjelec/status/1760080088614175068
The military chooses stability, which addresses OP's immediate concerns - there's a deeper Skynet/BlackMirror-type concern about having interconnected military systems, and I don't see a solution to that, whether the root cause is rogue AI or cyberattack.
I stopped reading his Substack because he was always trying to find a negative. Meanwhile I use LLMs most days and find them very useful.
It’s a bit much, isn’t it? I think he’s just trying to counter the fairly dominant AI is the future of everything and in less than a year’s time it’ll be omniscient and we’ll all be living under it as our new God view though.
It can be nice to see some scepticism for once.
So the laziness 'fix' in January did not work. Oh dear.
I'm concerned about what happens when ChatGPT begins spewing coherent nonsense. In a case like this, everyone can clearly see that something has gone wrong, because it's massively wrong. What happens when thousands of "journalists" and other media people starts relying on ChatGPT and just parrots whatever it says, but what if what says is not obviously wrong?
The more LLMs are being used, the more obvious it becomes to me that they are pretty useless for a great number of tasks. Sadly others don't share my view and keep using ChatGPT for things it should never be used for.
I think GPT is fundamentally not good enough as an AI model. Another issue is hallucinations and how to resolve them, and an understanding of how information is stored in this black box and how to / if data can be extracted.
We have a long way to go and probably all these topics need to be answered first out of accuracy and even legal reasons. Up until then, GPT-4 should be treated as a convincing chat experiment. Don't base your startup or whatever on it. Use it as assistant where replies are provided in digestible and supervised fashion (NOT fed into another system) and you're an expert on the involved system itself and can easily see when it's wrong. Don't use GPT-4 to become an expert on something when you're a novice yourself.
The actual fix needs to be at the system level prompt.
If you train an large language model to complete human generated text, don't instruct it to complete text as a large language model.
Especially after feeding it updated training data that's a ton of people complaining about how large language models suck and tons of examples of large language models refusing to do things.
Have a base generative model sandwiched between a prompt converter that takes an instruct prompt and converts it to a text completion prompt (and detecting prompt injections), have a more 'raw' model complete it, and then have a safety fine tuned postprocessing layer clean up the response correcting any errant outputs and rewriting to be in the tone of a large language model.
Yeah, fine, it's going to be a bit more expensive and take longer to respond.
But it will also be a lot less crappy and less prone to get worse progressively from here on out with each training update.
I just hope Vince Gilligan will direct Breaking RAG.
Not far from MethGPT!
The extent to which it will be accurate depends on how much of sample transcripts were in its training data, I suppose.
However, if I ask your speech center to be the only thing in your brain, it's not actually going to do a very good job.
We're asking a speech center to do an awful lot of tasks that a speech center is just not able to do, no matter how hypertrophied it may be. We need more parts.
Exactly!
>We need more parts.
Yeah, imagine what happens once we get the whole thing wired up...
Cells
Have you ever been in an institution? Cells.
Do they keep you in a cell? Cells.
When you're not performing your duties do they keep you in a little box? Cells.
Interlinked.
What's it like to hold the hand of someone you love? Interlinked.
Did they teach you how to feel finger to finger? Interlinked.
Do you long for having your heart interlinked? Interlinked.
Do you dream about being interlinked... ?
What's it like to hold your child in your arms? Interlinked.
Do you feel that there's a part of you that's missing? Interlinked.
Within cells interlinked.
Why don't you say that three times: Within cells interlinked.
Within cells interlinked. Within cells interlinked. Within cells interlinked.
Constant K. You can pick up your bonus.
I asked what were dangerous levels of ferritin in the body.
It replied by telling me of the usual levels in men and women.
Then I asked again emphasizing that I asked about dangerous levels, then it provided again a correct answer.
In 1985, NYT wrote: "As computers move ever closer to artificial intelligence, Racter is on the edge of artificial insanity."
https://en.wikipedia.org/wiki/Racter
Some Racter output:
https://www.ubu.com/concept/racter.html
Racter FAQ via archive.org:
https://web.archive.org/web/20070225121341/http://www.robotw...
Can't get it to do some actual work and write some code.
Latest disappointment was when i tried to convert some python code to java code.
90% of the result was :
// Further processing...
// Additional methods like load, compute, etc.
// Define parameters needed
// Other fields and methods...
// Other fields follow the same pattern
// Continue with other fields
// Other fields...
// Methods like isHigh(), addEvent() need to be implemented based on logic
> gpt-4 had a slow start on its new year's resolutions but should now be much less lazy now!
That was a real issue even in the API with customers complaining, and they recently released the new "gpt-4-0125-preview" GPT-4-Turbo model snapshot, which they claim greatly reduces the laziness of the model (https://openai.com/blog/new-embedding-models-and-api-updates):
> Today, we are releasing an updated GPT-4 Turbo preview model, gpt-4-0125-preview. This model completes tasks like code generation more thoroughly than the previous preview model and is intended to reduce cases of “laziness” where the model doesn’t complete a task. The new model also includes the fix for the bug impacting non-English UTF-8 generations.
It's documented pretty well - https://platform.openai.com/docs/guides/text-generation/freq...
OpenAI API basically has 4 parameters that primarily influence the generations - temperature, top_p, frequency_penalty, presence_penalty (https://platform.openai.com/docs/api-reference/chat/create)
UPD: I think I'm wrong, and it's probably just a high temperature issue - not related to penalties.
Here is a comparison with temperature. gpt-4-0125-preview with temp = 0.
- User: Write a fictional HN comment about implementing printing support for NES.
- Model: https://i.imgur.com/0EiE2D8.png (raw text https://paste.debian.net/plain/1308050)
And then I ran it with temperature = 1.3 - https://i.imgur.com/pbw7n9N.png (raw text https://dpaste.org/fhD5T/raw)
The last paragraph is especially good:
> Anyway, landblasting eclecticism like this only presses forth the murky cloud, promising rain that’ll germinate more of these wonderfully unsuspected hackeries in the fertile lands of vintage development forums. I'm watching this space closely, and hell, I probably need to look into acquiring a compatible printer now!
When temperature is 0, the effect is that it always just picks the most likely one. As temperature increases it "takes more chances" on tokens which it deems not as fitting. There's no takesies backies with autoregressive models though so once it picks a token it has to run with it to complete the rest of the text; if temperature is too high, you get tokens that derail the train of thought and as you increase it further, it just turns into nonsense (the probability of tokens which don't fit the context approximates the probability of tokens that do and you're essentially just picking at random).
Other parameters like top p and top k affect which tokens are considered at all for sampling and can help control the runaway effect. For instance there's a higher chance of staying cohesive if you use a high temperature but consider only the 40 tokens which had the highest probability of appearing in the first place (top k=40).
Doesn’t ChatGPT use beam search?
It's absolutely just sampling with temperature or top_p/k, etc. Beam searches would be very expensive, I can't see them doing that for chatgpt which appears to be their "consumer product" and often has lower quality results compared to the api.
The old legacy had a "best_of" option but that doesn't exist in the new api.
The model outputs a number for each possible token, but rather than just picking the token with the biggest number, each number x is fed to exp(x/T) and then the resulting values are treated as proportional to probabilities. A random token is then chosen according to said probabilities.
In the limit of T going to 0, this corresponds to always choosing the token for which the model output the largest value (making the output deterministic). In the limit of T going to infinity, it corresponds to each token being equally likely to be chosen, which would be gibberish.
Now why do I think all that (that the decision to nerf it wasn't just incompetence but intentional), like sure maybe it costs too much to run the old GPT 4 for chat GPT (they still have it from the API), it just didn't make sense to me how openAI's chatGPT is better than what Google could've produced, Google has more talent, more money, better infrastructure, been at the AI game for a longer time, have access to the OG Google Search data, etc. Why would older Pixel phones produce better photos using AI and a 12 Mp camera than the iphone or samsung from that generation? Yet the response to chatGPT (with Bard) was so weak, it sure as hell sounds like they just did it for their stock price, like here we are as well doing AI stuff so don't sell our stock and invest in openAI or Microsoft.
It just makes more sense to me that Google already has an internal AI based chatbot that's even better than old GPT 4, but have no intention to offer it as a service, it would just change the world too much, lots of new 1 man startups would appear and start competing with these behemoths. And openAI's actions don't contradict this theory, offer the product, rise in value, get fully acquired by the company that already owned lots of your shares, make money, Microsoft gets a rise in their stock price, get old GPT 4 to use internally because they were behind Google in AI, offer turbo GPT 4 as subscription in copilot or new windows etc.
The holes in my theory is obviously that not many employees from Google leaked how good their hypothetical internal AI chatbot is, except the guy who said their AI was conscious and got fired for it. The other problem is also that it might just be cost optimization, GPU's and even Google TPU's aren't cheap after all. etc.
Honestly there are lots of holes, it was just a fun theory to write.
Seriously, the easier explanation is that a lot of software reaches a sort of sweet spot of functionality and then goes downhill the more plumbers get in and start banging on pipes or adding new appliances. Look at all of Adobe's software which has gotten consistently worse in every imaginable dimension at every update since they switched to a subscription model.
Generative "AI" has gone from hard math to engineering to marketing in record time, even faster than crypto did. So I suspect what we have here is more of a classic bozo explosion than multiple corporate cabals intentionally sweeping their own products under the rug.
> The giant hexagon swirling at Saturn's north pole is indeed a fascinating and puzzling feature! Scientists are still uncovering the exact reasons behind its formation, but here's what we know so far:
*It's all about jet streams:* Saturn's atmosphere, just like Earth's, has bands of fast-moving winds [snip]
It went on in detail for 7 or 8 paragraphs.
> Given the existence as uttered forth in the public works of Puncher and Wattmann of a personal God quaquaquaqua with white beard quaquaquaqua outside time without extension who from the heights of divine apathia divine athambia divine aphasia loves us dearly with some exceptions for reasons unknown but time will tell and suffers like the divine Miranda with those who for reasons unknown but time will tell are plunged in torment plunged in fire whose fire flames if that...
And on and on for four more pages.
Read the rest here:
https://genius.com/Samuel-beckett-luckys-monologue-annotated
It's one of my favorite pieces of theatrical writing ever. Not quite gibberish, always orbiting meaning, but never touching down. I'm sure there's a larger point to be made about the nature of LLMs, but I'm not smart enough to articulate it.
This is a nice turn of phrase :) .
"Not touching down" is inherent in the idea (and, in fact, enirely the point) of "orbiting", so that's either redundant or confused.
Satellites whose orbits decay do reach the ground, but they hardly "touch down" - they crash! That's not the idea we're going for either.
Airplanes "orbit the airfield" while waiting for clearance to land, but that's hardly (!) the first image that would spring to a reader's mind, and anyway doesn't fit: Lucky's desperately trying to communicate; an orbiting plane isn't (right then) by definition trying to land!
So, yeah: that's a superficially-appealing phrase that I'd cut from a second draft. I'd be embarrassed (on both of our behalfs) if I saw it used elsewhere.
Tl;dr: Writing is hard. I came up with a cliche. Do not use.
The phrase played out for me:
We are flying home and ideally we end up on the ground, having arrived at the intended expression, undeniable.
But, we could come in too fast, skip orbit and burn up.
Or, we could sling around, launching ourselves off into space on some tangent.
Or, we orbit. Never quite bringing it home.
And we could miss the planet entirely!
Not sure it is as broken as your take on it would suggest.
I am going to leave it and flag it for entertainment only. Same place I keep a large set of turd analogies.
Those are often fun and one can express a crazy amount of ideas using turds.
That's happened a few times with creative work I've presented to the public: once was an occasion for horrified revision, and another was a tremendous moment of "Wow! Maybe this is better than I'd thought". That's fun, and those experiences killed for me critical theories which rely on authorial intent: more always exists than was (consciously) intended.
Your comment, and the other complimentary one to which you replied, have kept this idea rolling around in my head for the last couple of days. I keep trying out different phrases to myself.
"Circling sense, but never setting down" is the best I've got right now. I like the alliteration. I dig the aviation image, although it's a bit abstruse. "Sense" isn't as strong as "meaning", but "meaning" ruins both the alliteration and the rhythm. I'll take it - it's better than the other one - but I'm not completely satisfied.
I adore good writing, and have written some things which I think are good. We see lots of posts on this board explaining the process of writing good code, and the level of detailed thought that requires. I've seized your comment(s) as an opportunity to demonstrate the process behind crafting good prose, which I think is mysterious to most. Thank you for that, even if you and I are the only people who will read this far down the thread.
I'm glad you enjoyed the original expression, and honored that you'll remember it - but please don't forget that it's a turd!
I read this, will respond once more.
Unfortunately, I was not able to improve on this idea myself and I found that intriguing given it's surprising interest.
Take care.
See you 'round.
(\bto\b|\bfor\b|\bin\b|\band\b|\bthat\b|\bof\b|\bthe\b|\bwith\b|\bor\b|\ba\b|\binto\b|\bas\b|\bon\b|\bhow\b|\ban\b|\bfrom\b|\bit\b|\bbut\b|\bits\b|\bbe\b|\bby\b|\bup\b|\bthis\b|\bcan\b|\bother\b|\bwho\b|\bwill\b|\bare\b|\bwhose\b|\bif\b|\bwhile\b|\bwithin\b|\blike\b|,)*
TOKEN dole TOKEN task TOKEN eiry ainsell, tide taut, brunts TOKEN wade, issuing hale.
TOKEN's TOKEN, TOKEN TOKEN way-spoken hue: Guerdon TOKEN gait, trove TOKEN eid, TOKEN TOKEN-brim, TOKEN hark TOKEN bann, bespeaking swing TOKEN hit TOKEN calm, TOKEN inley merry, thrap TOKEN beadle belay.
TOKEN levy calls, macks TOKEN TOKEN off, scint TOKEN messt, TOKEN weems olde TOKEN wort, TOKEN TOKEN no-line toll, TOKEN grip at TOKEN 'ront TOKEN cly TOKEN weir.
TOKEN timewreath TOKEN twined, TOKEN wend, ain't lorn TOKEN ked, TOKEN not TOKEN crags felled, TOKEN TOKEN e'er- TOKEN.
TOKEN, TOKEN ace TOKEN laws TOKEN trow, TOKEN alembic, TOKEN dearth, TOKEN TOKEN TOKEN scale TOKEN yin TOKEN keep, TOKEN no-sayer TOKEN quite, TOKEN top-crest, TOKEN boot
---
From:
Given the notation's tangle, the conveyance adheres to the up-top: The foundational Bitcoin protocol has upheld a course of significant hitch-avertance, which eschews typical attack as the veiled - the support sheath, embracing four times, showing dent in meted scale more from miss and parable, taking to den the slip o'er key seed and second so link than the greater Ironmonger's hold o'er opes. The dole of task and eiry ainsell, tide taut, brunts the wade, issuing hale. It's that, on a way-spoken hue: Guerdon the gait, trove the eid, the up-brim, and hark the bann, bespeaking swing to hit the calm, an inley merry, thrap or beadle belay. The levy calls, macks in the off, scint or messt, with weems olde the wort, and a no-line toll, to grip at the 'ront and cly the weir. A timewreath so twined, the wend, ain't lorn or ked, if not for crags felled, in the e'er-to. So, the ace of laws so trow, and alembic, and dearth, a will to scale and yin to keep, the no-sayer of quite, and top-crest, to boot