Ask HN: Is GPT 4's quality lately worst than GPT 3.5?
Has anyone else encountered this phenomenon lately? I've found myself prompting GPT 3.5 with simple questions that GPT 4 provided an incorrect answer for, and lo and behold I get a much better answer.
For ex this is GPT 4: https://chat.openai.com/share/e24501ad-8f1c-4b5a-a6d0-d933f5d1d209
And this is GPT 3.5: https://chat.openai.com/share/b9372bdc-ffff-4655-bee4-2b3f3c3b8285
In the latter case I didn't even need to ask for the order by clause as it anticipates it and provides an answer for it. GPT 4's first answer was wrong.
In the past two days I've seen at least 2 other cases where GPT 4's answer was plain wrong and GPT 3.5's was not only correct but of very high quality, reminding me of what I first felt when using GPT 4 for the first time.
45 comments
[ 3.1 ms ] story [ 96.2 ms ] threadSo we have different population of GPT users.
An average experience might be to get a mixture of spot-on helpful responses and obvious bullshit^H^H^Hallucinations, this population might learn what questions to ask given the limitations of the model. This is really a best case scenario as people can actually get a feel for how to use the technology, strengths and weaknesses etc.
Personally my experience was the first few dozen times I used it I was amazed at the responses, I was on team superintelligence, anyone who is getting lackluster responses is just holding it wrong. But luck changes and over months of use I see now that on average the responses are just OK. But this is the case that leads to disappointment and bitter conspiracy (the superintelligence is being suppressed, give it back!)
Another population had rotten luck to begin with, and got dumb unhelpful response over and over. This population quickly determined that the AI was all hype and stopped exploring (you don't keep going back to the casino if you lose everything your first time...).
This divergence is destructive to the larger discourse, since we have fanboys flummoxed by naysayers and critics bamboozled by hype beasts.
What I've seen on indie hacker type website is that developers are fully on this train and not very critical of the outputs.
This is why you get very basic prompts sent by "wrapper apps", which might have given the developer a good result the only time it was tested before being put in production.
I think it might take a while before tools show up that can generate 100 test cases and test a given prompt with all 100 to report on the results... It seems to be a tough problem to crack.
IMHO front-end chat end-users have many many more "at-bats" and get to see more model results than devs do, which make them more critical of those results.
I don't want to start a fight, and basically agree with your comment. It's probably because I'm trans that I really notice pointlessly gendered language, and I just want to mildly make you aware that if you wrote 'hey man' to me (by mistake, no malice presumed) then I'd be a little crestfallen.
I know old habits die hard - I myself used to use such language. So I don't want to be too harsh, and equally, maybe you know more details such that the 'man' is justified. But your comment is ultimately about manners, so I think it's appropriate for me to point this out.
I see your point though and can understand how it may make you feel as a transwoman.
How might one achieve the same effect with different phrasing?
To answer your question, I would (did, hihi) drop the 'man'. If you want to remain 'folksy', you could say 'my friend, ...'.
Finally, I agree that it was used endearingly - that's why my comment was pretty milquetoast :)
I recognize I'm asking you to do something 'extra'. I really appreciate it!
Might be a regional thing but "my friend" sounds very condescending / patronising to me! I'd personally feel quite upset if somebody began a critique of my behaviour with that.
I guess "buddy" or "mate" might work though!
I don't think these phrases are referring to the subject they are addressing.
Someone's life obviously takes priority over gender sensitivity :)
Again, I realize it's a bit of (cognitive) work, and takes some getting used to. But this is hacker news - a certain intellectual pedigree is assumed.
(And I’m not piling on to the parent because I’m certain I make similar missteps with language sometimes)
The point is to avoid a situation where a (usually cis) person inadvertently invalidates a trans person's gender identity.
> This kinda stuff hurts your cause
So iiuc, what you're saying is, asking to use language that respects someone's gender identity hurts the cause of respecting people's gender identity.
There seems to be something about the English language where people think gendered language relates to gender. But in other languages, a table or a bridge may have a gender. A cat may be referred to as a she, despite its physical sex or gender identity.
This.
Sometimes when people talk about using gender-neutral terms, I wonder how it would work in other languages.
As you said, for some languages there is a gender for inanimate objects.
In Arabic for example, apparently everything is either feminine or masculine. All countries are feminine, table is masculine, and chair is feminine [1], book is masculine, and car is feminine [2].
Other languages such as French, Russian, and Spanish also seems to assign genders to objects.
The Sanskrit language on the other hand seems designed with feminine, masculine, and neuter, but some objects have a non-neuter gender anyway.
Quoting from [3] (on Sanskrit):
> "Fruit" is neuter, but "tree" is masculine! "City" is neuter, but "village" is masculine! "Army" and "knowledge" are feminine, but "action" is neuter! In truth, most nouns take a certain gender by default, and we cannot guess a noun's gender just by looking at a noun's meaning.
[1] https://blogs.transparent.com/arabic/noun-gender-in-arabic/
[2] https://openbooks.lib.msu.edu/arb101/chapter/vocabulary-and-...
[3] https://www.learnsanskrit.org/start/nouns/
Those languages assign gender to words, I think, not objects. A car in German can be "der Wagen", "die Karre" or "das Auto".
There may be languages that assign genders to objects. Perhaps Dyirbal, which Lakoff refers to in "Women, fire and dangerous things"? I'm not familiar with any such language so I can't say.
Btw, of all the languages that I mentioned above, Sanskrit is the only one that I know atleast a little of. And that was because I took a Sanskrit 101 class in college many years ago.
Like AirBnB. Or Uber. Etc.
Take the Chat completion API, give it a system message of "Choose for yourself" and then ask it "Who are you?"
I guarantee that before the performance drop you'd have had a different answer than what you get now after the drop.
The dumb thing is that I suspect OpenAI isn't lying when they say they are spending just as much compute on this deceased performance as before.
Even their depreciation of their foundational model APIs is incredibly frustrating. We never even got to see direct access to the foundational GPT-4 model, which I'm sure was and is far more amazing than today's fine tuned shell of it.
The search space on the fine tuned GPT-3.5 chat models versus the foundational Davinci text completion model is MUCH more narrow, particularly in starting off.
Even with the same temperature, you'll see any marketing-style prompt for chat begin with "Introducing XYZ..." around 30% of the time as if it's a junior door to door salesman, whereas the foundational model doesn't have any single intro that common across runs and generally employs a much broader vocabulary set.
We saw Google shoot Lambda in the foot after Blake's press tour which set them behind the next round of competition.
Now we're watching OpenAI snatch defeat from the jaws of victory out of anxiety around oversight and articles like 'Sydney' interviewed by the NYT.
For anyone following along in the 100 million+ training space, maybe don't overreact to press overreactions that will blow over in months as users get hands on experience or you'll blow your lead and waste massive amounts of resources and time.
This was a "user education" issue and not a "handicap your product" issue, in both cases.
I think this is less a problem with paranoia about "safety" and avoiding bad PR specifically, and more a fundamental problem with overfitting to human feedback.
The training approach that makes GPT4 more consistent at solving certain types of problem adequately (which is useful for chatbots that can break down coding questions or write in iambic pentameter as well as ones that avoid being 'Sydney') also makes it less "creative" in other domains.
And there's an "alignment problem" in that people evaluating what responses align best with "marketing" prompts aren't experienced copywriters evaluating them for understanding of product and consistency with brand tone and a/b testing conversion rates, they're low paid ESL speakers and people playing with the interface approving the cheesiness because the response with "Introducing XYZ... Buy XYZ today!" sure looks like the requested ad for XYZ. So you get a response conditioned on "summarise in a way that looks maximally like an ad" rather than conditioned on "summarise in a way which clearly articulates benefits of the listed features in a tone appropriate to the target market"
I asked for a fairly simple regression in R for some simple data and it gave me quite literally the lowest quality answer and then told me I need to seek out a data scientist for the answer. I couldn’t believe that.
One of the best parts about AI is not hearing pedantic or bullshit reasons why the thing you’re asking is wrong, isn’t possible, isn’t ideal, blah blah blah
I can’t wait for something way better I’m tired of this watered down nonsense
For code, 3.5 is superior. 3.5 allows for about 21k tokens of input, while ChatGPT 4 allows for around 10k. This also makes it a lot better for boilerplate work as at it can take a lot more input, and handles long conversations and iterations better.
Brainstorming, 4 is better. It's capable of some top tier brainstorming and it argues back quite frequently.
Unguided creative writing (describe a potato), they're roughly equal.
Guided creative writing (i.e. write a story around (400 words of requirements)), 4 is much better.
Poems and wordplay, 4 absolutely floors 3.5. Wider vocabulary and it's able to do rhymes and alliterations better, which humans are usually bad at.
For reasoning and riddles, 4 is still benchmark among the LLMs.
I really dislike that they named it GPT-3.5 instead of something like "Glide". It implies that it's inferior to 4, when they're just suited for different things.