HN Explorer

21 comments

[ 3.3 ms ] story [ 38.8 ms ] thread
[deleted]
(comment deleted)
woeirua
Interesting topic, but I'm not opening a PDF from some random website. Post a summary of the paper or the key findings here first.
hypoxia
Did you try it with high reasoning effort?
username135
I wonder what changed with the models that created regression?
oezi
There is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.
aresant
Feels like a mixed bag vs regression?

eg - GPT-5 beats GPT-4 on factual recall + reasoning (HeadQA, Medbullets, MedCalc).

But then slips on structured queries (EHRSQL), fairness (RaceBias), evidence QA (PubMedQA).

Hallucination resistance better but only modestly.

Latency seems uneven (maybe more testing?) faster on long tasks, slower on short ones.

fertrevino
Mixed results indeed. While it leads the benchmark in two question types, it falls short in others which results in the overall slight regression.
[deleted]
(comment deleted)
credit_guy
Here's my experience: for some coding tasks where GPT 4.1, Claude Sonnet 4, Gemini 2.5 Pro were just spinning for hours and hours and getting nowhere, GPT 5 just did the job without a fuss. So, I switched immediately to GPT 5, and never looked back. Or at least I never looked back until I found out that my company has some Copilot limits for premium models and I blew through the limit. So now I keep my context small, use GPT 5 mini when possible, and when it's not working I move to the full GPT 5. Strangely, it feels like GPT 5 mini can corrupt the full GPT 5, so sometimes I need to go back to Sonnet 4 to get unstuck. To each their own, but I consider GPT 5 a fairly bit move forward in the space of coding assistants.
agos
any thread on HN about AI (there's constantly at least one in homepage nowadays) goes like this:

"in my experience [x model] one shots everything and [y model] stumbles and fumbles like a drunkard", for _any_ combination of X and Y.

I get the idea of sharing what's working and what's not, but at this point it's clear that there are more factors to using these with success and it's hard to replicate other people's successful workflows.

kumarvvr
I have an issue with the words "understanding", "reasoning", etc when talking about LLMs.

Are they really understanding, or putting out a stream of probabilities?

causality0
I've definitely seen some unexpected behavior from gpt5. For example, it will tell me my query is banned and then give me a full answer anyway.
ancorevard
so since reasoning_effort is not discussed anywhere, I assume you used the default which is "medium"?
energy123
Also, were tool calls allowed? The point of reasoning models is to delete the facts so finite capacity goes towards the dense reasoning engine rather than recall, with the facts sitting elsewhere.
[deleted]
(comment deleted)
mattwad
i thought cursor was getting really bad, then i found out i was on a gpt 5 trial. gonna stick with claude :)
0xDEAFBEAD
So which of these benchmarks are most relevant for an ordinary user who wants to talk to AI about their health issues?

I'm guessing HeadQA, Medbullets, MedHallu, and perhaps PubMedQA? (Seems to me that "unsupported speculation" could be a good thing for a patient who has yet to receive a diagnosis...)

Maybe in practice it's better to look at RAG benchmarks, since a lot of AI tools will search online for information before giving you an answer anyways? (Memorization of info would matter less in that scenario)

andai
Did this use reasoning or not? GPT-5 with Minimal reasoning does roughly the same as 4o on benchmarks.