75 comments

[ 3.1 ms ] story [ 148 ms ] thread
"Materially better" how apt and rather poetic. All we seem to care about these days is the material.
I mean when you pay for a tool, yeah
So when you upgrade something you don't look for it to be better? I don't understand this comment.
He is salty and pessimistic about the future.
Yes, but more accurately, never mind the future: I am pessimistic about the present.
Something materially better in the sense of efficiency and short-term gain is not necessarily something better. Like the car: yes, it was faster than the horse, but now we have climate disaster. AI is another example: the upgrades are shiny but ultimately harmful.
I wonder if now we enter the predictable loop of updates that make things ‘marginally better’ for some while worse (basically the same quality but a price hike) or the same for most.
Until they can figure out how to scale other modalities to the expected amount of users , Incremental increases in Text models is likely what we will be seeing for the next little bit
Could be, but this has not been happening with openai. Their upgrades are legit. Thus There's no trendline to indicate this is what will happen..
Any news on Q* ?

On a sidenote, my first thought was someone was a StarTrek fan.

Altman said in an interview last week with Lex Friedmann it‘s about reasoning, they are working on it, but have not yet solved the problem
Given that GPT-4 was released a year ago now, and given that GPT-4 is in many ways 'not there yet', GPT-5 will need to show a big material improvement. Anything less will signal the end of the road for LLMs.
GPT-4 is already hugely useful. If they are able to lower the cost of GPT-4 further, say 5x or 10x that in itself would be useful and huge.
IMO that's the key there. I think a lot of people who make comments like the GP have either just been using the chat interfaces or perhaps have not gone deep with implementation on some of these models.

If you could both speed up inference time and reduce cost of GPT to 3.5 levels, there are incredible amount of possibilities that open up that could actually help solve a lot of problems have had with trying to interface software to the real world (robotics).

4-Turbo is actually pretty amazing but its still a tad pricey for some tasks. Altman made a comment on Lex's podcast about compute and I think its true, essentially the world does not understand how much compute will be desired as the price of these things go down.

Agreed, the chat interface is quite limited compared to what's possible with the API. I recently wrote a script for my job at a publishing company that automatically writes social media promos for every new article published on our site by crawling the sitemap on a cron job. Still putting the finishing touches on it, but hoping to implement it soon.

But with only the chat interface, our social media manager would have to find every new article in every discrete vertical on our site, copy all the URLs to a spreadsheet, then prompt ChatGPT for every article, wait for it to respond, then copy that response into the spreadsheet...

> If you could both speed up inference time and reduce cost of GPT to 3.5 levels

Presumably this would be a major part of the big material improvement we'd need to see.

Tangentially, I lowered my personal cost by more than 3x, and was able to share GPT-4 with my friends and family.

I installed LibreChat on a Linode droplet, put $10 on my OpenAI account for API usage, and cancelled my ChatGPT subscription. Since neither I, nor my friends and family use ChatGPT a ton, I let them sign up on my server and the API costs are under $1 so far.

I actually think this is the way. ChatGPT honestly is not that great. I would be happy to pay per token for ChatGPT and be able to have a little more control over the system messages.
If you regularly use GPT4 for coding this isn't so viable, it gets pricey with the amount of context you need and the default GPT4 chatbot is decent with code. The GPT API is best hit with questions that don't have a lot of setup, can leverage its excellent general knowledge, and can be answered succinctly.
I find it pretty viable for myself. I use this kind of setup within the IDE using API keys. I will flip between 3.5 and 4. The cost of 4 is minimal compared to my time.
This sounds like a great idea and relevant to my current goals. I haven’t used LibreChat; how are you handling authentication?
I just let them sign up with an email and password, then I turned off the ability to sign up once everyone had done that. LibreChat does support other options [0].

I configured the yaml file to use my own OpenAI API key for all users.

[0] https://docs.librechat.ai/install/configuration/user_auth_sy...

edit: oh, the only other configuration change that I made was to set the default OpenAI model to gpt-4-0125-preview.

Got it, thanks! I’ve been wanting to do some API work anyway, and this seems like a great way to consolidate everything.
What do you mean by "not there yet"?

I regularly use GPT-4 for things like "write exhaustive unit tests for the following function" and it gives fantastic results. They usually need fleshed out a little, sure, but in a few seconds I get something that would have take about $45 in billable human hours.

I personally cannot wait to see the productivity gains that come from GPT-5.

> They usually need fleshed out a little, sure,...

> I personally cannot wait...

My point exactly.

Would you be disappointed if someone came out with a magical new paintbrush that could paint any color in oil, acrylic or watercolor without actual paint because the paintbrush didn't levitate itself and do the painting for you?

You can get red in the face at AI the disembodied futurist concept all you want, but the tools are the real deal - people are using image generation and chatbots today to be 50%+ more productive and produce better work. These tools require skill to use, they're not just plug-n-play, but that skill pays huge dividends, and if you don't learn it you're going to be left so far behind.

> people are using image generation and chatbots today to be 50%+ more productive and produce better work.

I don't think we have evidence to support that.

From the abstract, it is suggested that for some tasks AI was a benefit, but for others it actually made things worse.

So can we know which are which, and what are the relative value adds of these task groups? Where does an overall cost benefit analysis land, bearing in mind the high costs of AI.

Easy, AI is a tool like a hammer. Know when to use a hammer, vs when to use a saw or a drill? Models are a more abstract tool but if you spend time with them you'll build intuition quickly enough.
I’m already completely bored with 4, so can’t wait to see what 5 brings.
How quickly humans seek more and more. A year ago GPT-4 was earth shattering and now people are bored with it.
> A year ago GPT-4 was earth shattering

No it was not, and nor were its predecessors.

The only people who think GPT is "earth shattering" are rose-tinted-spectacle AI dreamers, and the world's media who love a good AI story.

Anyone sensible can see that GPT is merely the latest iteration in "AI will rule the world" nonsense. It comes round every 5–10 years, I remember most of the previous hyped up iterations, this one is no different, its just the hype is sadly sticker this time round. Its still hype though.

I work in the industry and not sure what you do but 5-10 years ago there was definitely "AI" and lots of hype in ML but the output potential was no where what it is today. GPT-4 is indeed earth shattering in what you can now automate.
> GPT-4 is indeed earth shattering in what you can now automate

Just because you can, doesn't mean you should.

Companies using AI to automate "customer service" ? I encounter that regularly in my daily life these days. Its shit. The "intellligent" robot is always a useless waste of time.

I understand your frustration but I suspect you are interfacing with some of the older style systems and not one built on top of GPT4.

I think in cases of new LLM implementations you can and should.

How? I mean my “wonder”-factor has probably decreased but it’s an integral tool for me.

CLI one-liners or simple scripts are probably my main use-case. So many things I always knew were possible (and implemented in some cases in the past) but often avoided since I wasn’t sure if it would pay off that now I can do with ChatGPT. Simple things like “benchmark this endpoint and output the min/max/avg/p99 stats every 10 requests with a rolling window of 100 requests”, no it’s not hard to write that but often I just wouldn’t have to time to test out a theory so I’d skip this.

I’ve lost count (easily over 30+) times I’ve dealt with a production issue using ChatGPT to write something to better analyze the problem (parsing stuff out of logs often). A whole slew of “I wonder if…” questions that I’m able to answer in <2min vs having to spend 10-15+ if I did it all manually.

No, it’s not perfect and sometimes I need to go back and forth a few times but the overall speed up using ChatGPT is significant.

> how?

They nerfed it at Dev Day in November. It’s just no longer the magic tool it was before than anymore. It’s ok for maybe one shot problems or refactors of small functions but the version where you could in an hour work back and forth with it and build something moderately complex from scratch is dead and often some of the generations just waste time rather than solve the problem.

Is there real proof of this? I’d love to read more about it because I’ve been so frustrated the last few months.
I don't know that I'd say they nerfed it exactly. They made some tradeoffs you don't like. The old GPT4 had more personality, was more flexible in its responses and would go on for longer. The new GPT4 is more consistently helpful and does a better job of presenting small-form answers neatly, but the alignment has definitely had an impact for a lot of use cases.
Which translates to: GPT-5 / next version is an incremental update.
Which leads one to guess where we now are on this particular sigmoid.
(comment deleted)
Is this the same "materially better" that the present version gives you when you tell it the auto-generated code it produced for you is a pile of shit, and it presents you with a new version that it promises "fixes the problems", but instead introduces more.

AI is and always will be supremely confident at being supremely wrong. Its the nature of the beast. Some things are just best left to real-life humans.

I mean, it imitates humans. Have you met humans?

If you want something real smart, what you need is a Casio calculator.

Humans, in isolation, also tend to be supremely confident at being supremely wrong. Having others around and hearing their criticism of our opinions is a crucial factor of converging towards the truth.

Perhaps a system of mutually-criticising LLMs is what is needed to avoid confidently incorrect answers? Give them all a task to find errors in each others' outputs, and only return the output to the user when all of them agree on correctness.

> Humans, in isolation, tend to also be supremely confident at being supremely wrong

At least most humans will know that most modern programming languages have built-in basic math functionality.

When I asked GPT-4 to write me a basic program in Go, it tried to tell me that a basic math function was not available in Go stdlib and proceeded to write some DIY horror-show of code as a substitute.

On the same occasion it also took great delight in importing obsolete libraries, and using deprecated functions.

And for the icing on the cake, the code it generated did not even compile !

Every time I corrected it, it returned with fresh code and a supremely confident assertion that "this fixes the problems identified". It was wrong, of course. Every, single, time.

All basic errors that even a Junior programmer fresh out of school would not make.

What was the basic program you asked for?

Hard to evaluate if this is a fair test without knowing what you asked of it.

> Hard to evaluate if this is a fair test without knowing what you asked of it.

Oh, right, the old "can't possibly be the AI, must be the user" claim.

The same excuse certain car manufactuers use when their car gets confused by shadows in the sun.

Give me a break.

I grow skeptical of these angry anti-AI posts when they lack such easily-provided evidence. There is a share button on every chat on ChatGPT. You could just share the conversation here and spare us all a lot of pointless back-and-forth.

https://chat.openai.com/share/371863ec-edbd-4454-8b19-382035...

Here is an example of the kind of scripting I do regularly with ChatGPT. I cannot speak to its capabilities with Go, but it is quite proficient at Python.

OK, but (I mean this sincerely and in kindness) this isn’t a particularly challenging task. Useful, yes, and the type of thing Python is marvelous at, but not overly difficult.

If you start getting into more esoteric edges of Python, like SharedMemory, ChatGPT quickly falls apart. Or, more relevant to your example, using carriage return to overwrite text for a progress indicator – works great, very simple. Until you try to use it through subprocess. To ChatGPT’s credit, it eventually came up with using pty, which with some massaging, I got to work.

I’m not saying it isn’t useful – far from it. It’s just that on anything modestly complicated, you sometimes have to spend more time fiddling with the prompt than if you just sat down and wrote the code. Another example that comes to mind was implementing a B+tree in pure Python. I know how they work, and wanted to see if ChatGPT could figure it out. You’d think so, right? But no, it kept getting stuck on node splits.

Zero offense taken, and I agree completely, it's just that most tasks a person would want to do with Python are probably simple enough to where usage of a language model will save a significant amount of time.

It doesn't mean you can turn your brain off and just mindlessly copy and paste whatever gets output, but for basics and boilerplate it's a massive time save, and I only foresee the capabilities getting better over time. It's just that comments like the one I was replying to come off as childishly throwing one's toys out of the pram because they are imperfect.

Current generation LLM's aren't about doing tasks that humans struggle with - they are about doing tasks that humans find easy quicker than humans can do themselves.

Future generation AI's are highly likely to have more capability though - I wouldn't bet the house on the current state being the future state.

> most humans will know that most modern programming languages have built-in basic math functionality

I doubt that most humans will know what those words even mean.

But pedantry aside, what I was trying to say is that people, without being corrected by others or without having self-criticism embedded in their way of thinking, make same mistakes as LLMs do. Think of an isolated programming novice who lives in his mom's basement and whose knowledge of programming comes only from stack overflow, random blogs and documentation he doesn't really understand. He'd probably give the same output as an LLM.

Considering the way LLMs work, I find them conceptually most similar to a stream-of-thought kind of thinking - which, in human cognition, is only the first step in reasoning. The other steps are observing what has been thought, detecting mistakes, fixing them, and doing so repeatedly until the thought is deemed to be correct.

I'm just saying maybe it's possible to emulate (to some level) that process of reasoning by connecting multiple LLMs together and giving them a task to criticize and correct each other.

But then again, you don't seem to be even acknowledging that point of mine. Maybe you're just angry/jaded/<insert negative emotion> at LLMs and feel the need to vent your frustration. And then again, maybe not. If you are, that's understandable, but I think that HN is not an appropriate arena for that - Twitter or 4chan would be more appropriate.

> AI is and always will be supremely confident at being supremely wrong. Its the nature of the beast. Some things are just best left to real-life humans.

Sounds like you are also supremely confident at making predictions!

AI's are already less confident and more cautious than their previous iterations, so I'm not sure how you can confidently conclude it's entirely unsolvable.

> Sounds like you are also supremely confident at making predictions!

This is not my first rodeo as they say.

I've seen it before. I've seen incredibly intelligent people before going all-in on the latest AI hype. And it always ends the same way, badly. The hype dies down and then returns again when the next iteration comes around.

"I don't think it worked in the past, therefore it can't work in the future" isn't a particularly compelling argument considering the number of breakthroughs in the last 3-5 years.
Its funny, you see comments like this and I think people confidently give very specific hurdles to them for what these models should be able to do.

I have a rather large spend across the universe of models and I think compared to a year ago, its amazing what is possible. If we continue anywhere close to this speed, it will be amazing what will be possible a year from now.

> If we continue anywhere close to this speed, it will be amazing what will be possible a year from now.

People said that a year ago and look today, still gpt-4 but now its turbo. So if we keep that pace in a year we will be at roughly gpt-4 but turbo++, better but not a giant leap.

Massive cost decreases. Price increases. Forced json. Seems pretty incredible to me. Gpt4 on release a year ago was completely different.
Agreed. Algorithmic slight of hand. I'm getting tired of these toys.
It's easy to confuse text generation with thinking.

I found that if I treat it as a program that can generate text, it's a lot easier to write the prompt that generates code that's more or less what I need.

It can do the boring stuff pretty easily, like generate boilerplate code or settings files. It can explain technical issues better than googling.

It also helps that I use a functional language with immutable data structures and the code it generates is either good or good enough to give me an idea on how to move forward.

Once you lower your expectations and use it for what it is, it's a very powerful tool.

I wouldn't say always. The way the "confidently wrong" answers make sense to me are in the context of asking people trick questions that seem to have an obvious answer but actually require more deliberate thought to get right. What happens in that case is you are engaging the part of the brain that has simply memorized a fact or developed a quick but not always accurate heuristic which it then applies to the question that seems to meet the criteria and it results in a confidently wrong answer given by a real meat and bones human. LLMs right now are essentially a big collection of facts and heuristics with the ability to find the often right association between input and the applicable output. Everyone is working on building the other thing that does the careful step by step reasoning and error checking, the so called System 2, and if brains can tell us anything about the implementation of such a system it is that it's made of the same stuff that System 1 is but arranged and controlled in such a way that it produces more reliable answers.

*edit: I should specify that when I say LLMs are a collection of facts and heuristics I mean they are a collection of those things encoded as language which itself has been encoded as vectors of floats which in turn have modified the weights of the network to produce yet another encoding. I don't mean that the facts and heuristics are stored in a lookup table or as procedures.

Clickbait title and contradicted by the sama quote:

> We’ll release in the coming months many different things. I think that’d be very cool. I think before we talk about a GPT-5-like model called that, or not called that, or a little bit worse or a little bit better than what you’d expect from a GPT-5, I think we have a lot of other important things to release first.

In other words, the next OpenAI GPT update(s) WILL NOT be GPT-5. Yes, we can expect improvements (because of course) but we shouldn't expect a major leap forward in AI the next couple of months from OpenAI.

When a CEO says "before we can talk about" it means "no". The language about other cool updates suggests incremental improvements or platform improvements, but not a major leap forward in terms of intelligence.

In the latest Lex Fridman interview he said:

>We will release an amazing new model this year. I don’t know what we’ll call it.

And:

>So part of the reason that we deploy the way we do, we call it iterative deployment, rather than go build in secret until we got all the way to GPT-5, we decided to talk about GPT-1, 2, 3, and 4. And part of the reason there is I think AI and surprise don’t go together. And also the world, people, institutions, whatever you want to call it, need time to adapt and think about these things.

>We’ll release in the coming months many different things. I think that’d be very cool. I think before we talk about a GPT-5-like model called that, or not called that, or a little bit worse or a little bit better than what you’d expect from a GPT-5, I think we have a lot of other important things to release first.

https://lexfridman.com/sam-altman-2-transcript

From the phrasing of the second quote it seems like GPT-5 might be synonymous with their conception of AGI or at least agentic AI. Maybe it was just bad phrasing and he was using GPT-5 as a stand in for the ultimate goal but to me at least it seems like they're holding back on it because they're still trying to get the agentic and reasoning parts right. I think I agree with his musings about Lex's reaction that maybe they aren't releasing iterations fast enough for them not to look like big leaps. I would think they'd have something by now that could at least be labeled GPT-4.5 and be deployed as an incremental improvement. Maybe now with Claude 3 Opus on the scene they'll need to ship something soon even if it isn't a huge improvement.
IMHO none of this meaningfully contradicts the parent, and some of it frankly comes close to outright insulting his audience's intelligence.

> So part of the reason that we deploy the way we do, we call it iterative deployment, rather than go build in secret until we got all the way to GPT-5, we decided to talk about GPT-1, 2, 3, and 4. And part of the reason there is I think AI and surprise don’t go together.

This was presumably said with a straight face, which I find both impressive and unsettling.

When CEOs and politicians speak you have to listen to what they say but also listen for what they don't say. If OpenAI had GPT-5 almost ready to go, sama would have clearly hinted towards that. He wouldn't talk about a model being maybe "a little bit worse than". That's not the language you use when talking about the next leap forward in AI. When asked about GPT-5 he immediately deflects and talks about the "other cool things" they have in the pipeline.

When a CEO deflects when presented with an opportunity to hype the next major leap forward in AI that's not a good sign. He talks about "iterative development" to give "the world [...] time to adapt". He's telling people to temper their expectations.

This is how Sama has always been though. He talked down GPT-4 and that was still a huge leap over 3. If anything, he's been more hypey than for 5.
Any title with the word "might" isn't worth reading.
He was on Lex podcast again? Someone please tell me what that guy has on people. The ratio of success to skill is the biggest I have ever seen in pretty much everything.
sycophantic softball interviews, easy pr to influence techbros