Back in April it would only generate a handful of tokens per second. The speed improvements for GPT-4 are staggering. I wonder how much of it is because Microsoft is making GPUs rain on OpenAI, and how much of it is due to improvements to the model and its scaffolding.
I think the general (completely speculative and unconfirmed) consensus is that "Turbo" models are somehow quantized, or otherwise modified for much faster and cheaper inference, at the expense of some quality (the definition of "some" is unknown here).
well, that's how it was for GPT-3.5 anyway. The "turbo" flavor was faster and cheaper, but seemed to have slightly worse output (again, this is all going by subjective measurement; it could entirely be the imagination of AI bros)
I agree that this is the case for GPT-3.5, at least subjectively. However, with GPT-4 Turbo, it seems that performance has improved. If they got it to be faster using quantization, then they must have also found a way to offset any resulting performance losses.
Based on rumors, it seems that because the demand is so high and supply so short, that Nvidia is having to select who gets the cards. I bet they're likely thinking about who can do the most impactful work and OpenAI would definitely be in the running for the short list of companies who are actually shipping. So I bet OpenAI / Microsoft got a lot of the newest cards.
H100 + quantisation + algorithmic improvements would be sufficient to explain the speed boost.
If you "have enough compute" available -- which OpenAI definitely does -- the best current technique is to use mixed precision with post-quantisation fine tuning to restore performance. That's most probably how all of the "turbo" models work. Take a model that was initially 16 or 32 bits per parameter during training, quantise it down to a mixture of 4, 8, and 16 bits, and then fix it up with an additional training pass that uses the original full-fat model's predictions as the loss function. With access to the raw parameters, it's possible to do this training such that all of the output weights are considered and adjusted during this phase instead of just the top word. Third parties fine-tuning against GPT4 chats can't do this, even with the collected samples, because they only have individual selected tokens/words instead of the full probability distribution.
So it appears that GPT-4-Turbo is indeed (at least marginally) smarter than the previous GPT-4, just as Altman claimed. Also, it's faster and cheaper, with a massive context window. Exciting!
Yes I was really worried this was going to be "better" as in faster and cheaper but not "better" as in smarter. I've been playing with it for the past day and haven't noticed it any smarter per se, but also haven't noticed it dummer either.
Perhaps it may have worse reasoning for some tasks, so having "turbo" lets people know they can still try the old version if turbo doesn't work for them? Kind of like 3.5 vs 3.5 Turbo.
The increased context size will have the most significant impact on my work. That's where I run into limitations, when reviewing written work and code. I've been feeding written work into GPT in chunks, and I'm really happy to be able to feed in whole pieces. (I don't have it revise anything for me, I just have a specific prompt for exactly the kind of feedback I want on written work.)
I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
Have you tried aider-chat? It does some interesting things with tree-sitter so it can give the LLM a context (files, classes, functions, parameters, etc) as well as certain full files. That way it has your entire codebase in API form and it can focus on the actual code you are looking at or editing.
I haven't tried it yet, but people in the /r/chatgpt subreddit are claiming GPT-4-Turbo seems to have issues with understanding/remembering longer (say 100 lines) of code, whereas 3.5 and 4.0 seem to have handled things a bit better, implying that the context-window size isn't (currently) as large as claimed.
The context window IS longer, but it's less powerful. Obviously, they can't afford to have full transformer context over the entire context. That would be an impossibly large amount of ram. They're using some combination of sliding window/cyclical/or some other adjusted attention mechanism likely with some degree of summarization in some manner.
The progress here is remarkable. A year ago, we didn’t even have ChatGPT. LLM completions were cool but so hard to use and definitely there was nothing accessible to non-nerds.
You can try it out in the playground but not the consumer ChatGPT website. You will be limited to 10K tokens per minute if you are not in any spending tier e.g $150 per month will give you bigger context window.
That Reddit user mentions something that I have noticed happening with infuriatingly increasing frequency in the last few weeks (at least on the chat GPT interface) - the "fill in the rest here" or "remainder of code" snipped sections even when you EXPLICITLY instruct ChatGPT to include the fully modified code in its responses.
It's like the AI equivalent of "the rest is left as an exercise to the reader" you'd find in old textbooks.
This is so annoying. And it is getting worse. And a great way for OpenAI to hinder competition and then copy the products being developed using their APIs. This is openai cheating on their userbase.
Agreed, this is single handedly making me hate chatgpt for coding. I wish OpenAI would respect it's paying users enough to say what the changes and limitations are. (but i guess like google, i am not openai's customer. Their investors are. i am the product).
gpt4 should really be called DecartesGPT with this bs.
From the anecdotes there, it sounds a whole lot like something is going on where it's internally doing a summarizing step to fit into a much smaller actual context window than 128K.
The past days ChatGPT went from a great pair programming helper to a useless antipathetic intern, the quality of generated code dropped visibly. The context seems to be bigger in the chatgpt plus version too but it got dumber.
For other (non-code) benchmarks, people are having the opposite experience:
"I benchmarked on SAT reading, which is a nice human reference for reasoning ability. Took 3 sections (67 questions) from an official 2008-2009 test (2400 scale) and got the following results, here a SAT-like test:
Does anybody know if 2008-2009 SAT is in the training set for these models? Assuming so, I’d be especially interested in head-to-head evals on this type of non-code benchmark for problem sets not already in the training data, to see how it performs on fresh situations.
You seem to be suggesting it got a bit worse, and the aider article seems to suggest gpt4 got a bit worse, although much faster at being a bit worse, while gpt3.5 got worse, then better, while faster.
The Aider article has been updated with the complete results. Previously Turbo was leading slightly. So far any difference is in the noise.
However, in my opinion the first attempt score is more important, and Turbo does genuinely seem to lead there. There's still a possibility the updated training data has tainted the results.
Programmers here seem excited about the potential of this new version...but I can't help but wonder at how naive this attitude really is. Even if AI never becomes intelligent like us, if it can emulate this intelligence in enough domains, then it has a serious chance of being dangerous. It's already pretty much guaranteed that it will put almost everyone out of a job, turning the vast majority of humans into content-consuming sloths.
Does it really make sense to play with this kind of power?
It’s way too late to pretend AI won’t have an impact on programming as a profession.
Better to be excited and learn the tool as it develops than to stick your head in the sand.
A world where AI has put literally everyone “out of a job,” meanwhile, is still so far from our current reality that IMO, it’s not worth making practical day-to-day decisions on unless you are directly involved in the development or regulation of AI.
I understand your broader point, which is that progress in this space will be discontinuous and hard to see until it’s already happened.
But two years ago, GPT-3 was already astounding. ChatGPT made some pretty huge gains, and GPT4 more still, but the thing that exists today was relatively predictable, compared to a world where human effort is no longer important.
You are assuming that the whole existence of humanity is to work? because, without working, they would be sloths?
What about expending more time having healthy habits like working out, meeting more often with family and friends, discovering the world, learning new stuff?
So retired people are just sloths?
I’m more worried about people not being able to feed themselves because their labor became worthless. They will effectively be frozen out of the economy as they have nothing to trade with.
Frankly, I think eventually machines will do it all. I see AGI as the universal automation that can do everything a human can - apart from “being human”.
> How many working horses are there today vs before automobiles?
Well, exactly. The less working horses there are, the more expensive and exclusive it would be to ride them.
There could be 1 trillion automobiles and I bet you, none of these automobiles would compare to riding a real, live horse.
Similarly, there could be 1 trillion AI robots, they could do everything better than a human, and yet I bet you'd still want to ride (or otherwise experience) a real, live human.
My point is that if automobiles were always better than horses in every way, then nobody would want horses. But even today, with the amazing automobiles that we have, some of which even faster and more reliable than most horses, it's clear that we still want horses.
My question is, if horses were as intelligent as us and they could have their basic needs met extremely cheaply, would they be willing to work at all, apart from the occasional ride? Because the horse labor pool would shrink immensely if they didn't really want to work.
If AI does everything, the economic won't make sense anymore. Maybe there would be a basic rent or just anyone will ask for what they want and AI will provide it.
We though AI would replace the low level jobs first, but it seems creative jobs are gone first (art, software developers, etc). Bear that in mind.
No matter who gets replaced first, someone is getting screwed.
Frankly, if it's the higher end jobs getting replaced first that would likely spill over to the lower end ones as the those people who lost their jobs resort to taking lower end work to survive, flooding the market.
That's under the assumption that nothing else will change. But it is not the case, the system would have to adapt. One possibility is that we wont use money anymore, and there are a lot of in betweens in the middle. But for sure what you cannot do is to stop the change that is coming.
It bothers me a little bit as a programmer, but the rational part of my mind is aware that there's literally never been a major technological advancement in human history that didn't ultimately result in more people being employed, not less. Improvements in productivity just result in people realizing there's a whole new world of stuff they can now build that wasn't feasible before, and the market will eat that up.
The shape of my career might change, but I doubt I'll be unable to find a job.
Ask horses how they're doing these days... just because we've always found a use for humans because of their unique cognitive abilities doesn't mean we always will find a use.
I think it's critical to be thinking about how to make sure wealth isn't simply funneled to a few capitalists who own everything simply by virtue of them being first, because it seems that's the future we're heading for if we aren't careful.
I think you, and I, and everyone on HN will be fine (more or less...) but I am worried about a wide swath of people who will get "left behind."
"The [horse] population peaked in 1910 at 27.5 million, more than 600% higher than in 1840. What does the overall growth of horses and mules during the nineteenth century suggest about the relationship between animal power and the introduction of new mechanical technologies, including the steam engine?"
> there's literally never been a major technological advancement in human history that didn't ultimately result in more people being employed, not less.
In the past, new automation technologies often open up new possibilities in production capabilities in turn creating new jobs - specifically jobs that have not been automated yet.
AI though promises to be the universal automation, i.e. it can do any job. Thus even if new jobs show up, they will be taken over by AI too.
Then what?
> The shape of my career might change, but I doubt I'll be unable to find a job.
Question you should ask is why would anyone hire you when they can get AI to do the same job.
> Question you should ask is why would anyone hire you when they can get AI to do the same job.
My guess is that some people will always be stubborn and prefer me over an AI giving them massages every now and then, even if I won't do them as perfectly.
But at that point in time, I think me giving one massage would perhaps be enough for me to retire for life, as long as I can abstain from buying luxury goods or services, i.e. those provided manually by other people (instead of AIs, whose labor would be practically free).
> My guess is that some people will always be stubborn and prefer me over an AI giving them massages every now and then, even if I won't do them as perfectly.
But why you? There are billions of other people who are out of work they could hire for the “authentic human experience”.
> AI labor is effectively free at that point, didn't you get the memo? :)
It’s not. Machines need power to run. Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
Did you forget about all the endless solar panels and wind turbines that the AI robots progressively built over the years, for an increasingly cheaper price? Energy is abundant now.
They also built a bunch of AI robot factories, so now we have effectively limitless AI robot labor. Everyone has AI robot assistants/friends now.
> Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
But they have something to give in return. These owners want authentic human experiences, which can only be provided by other people, not AI robots.
Authentic human experiences (or hand-built goods) are thus now much scarcer, since most people don't need to work anymore. They're sort of luxury goods/services. In this world, imagine how much you'd be willing to pay to have an exclusive, authentic house maid or hairdresser for one day, rather than just another AI robot, no matter how good it is.
Or imagine dining at a restaurant where every waiter and every cook are authentic humans, wow! An unforgettable (and prohibitively expensive) experience.
Of course, not everyone could afford that, unless they themselves are willing to provide such a service. Most people would eat at home (with food farmed, prepared/cooked and delivered by the AI robots and drones, of course), or just go to regular AI robot restaurants if they're feeling especially fancy.
> I think you might be overvaluing “authentic human experiences”. It’s a luxury, not a necessity.
Rich people are notorious for seeking luxuries.
> There will also be an over supply of people willing to offer it because they have nothing else to offer in trade for basic necessities.
I don't know, perhaps you are right. I do hope you are wrong, though... :)
Also, I'm hoping if such an extreme situation would present itself, governments would be forced to step in and somehow ensure people would always have their basic needs met by, say, a strategic national AI army dedicated to provide those basic needs.
> I'm hoping if such an extreme situation would present itself, governments would be forced to step in and somehow ensure people would always have their basic needs met by, say, a strategic national AI army dedicated to provide those basic needs
>AI though promises to be the universal automation, i.e. it can do any job.
An AI can't do any job that requires a body, so clearly that isn't true.
If you mean mental jobs, this could be true in the future, but right now, it's hard to say. You can imagine scenarios where an AI does 95% of a job but a human is still required for the last 5% (without which the first 95% is useless).
These tools are recent, but not THAT recent. GPT3 is nearly 4 years old, Copilot is 2 years old, ChatGPT is a year old, GPT4 is 8 months old. No large-scale employment changes resulting from AI are visible so far.
> an AI does 95% of a job but a human is still required for the last 5%
This is the kind of scenario I was talking about in my initial comment.
The naive assumption about this scenario is that it would result in 95% of jobs disappearing, but based on history it's more likely that we just do 2000% more stuff with the same amount of human effort.
And if AI really replaces 100% of human labor, are we really going to retain an economic system that only works when humans are doing labor?
Analogous viewpoint in the Eighteenth Century: "90% of the human race consists of agricultural laborers, mostly unfree (slaves or serfs). Steam-powered machinery will put almost all of those people out of work, turning the vast majority of humans into food-consuming sloths".
Agriculture's share of the labor force went from something like 70% in the early 20th century to something like 5% now. You have a higher purpose than doing a robot's job badly -- or a farmer's. Find it.
94 comments
[ 2.8 ms ] story [ 171 ms ] threadwell, that's how it was for GPT-3.5 anyway. The "turbo" flavor was faster and cheaper, but seemed to have slightly worse output (again, this is all going by subjective measurement; it could entirely be the imagination of AI bros)
If you "have enough compute" available -- which OpenAI definitely does -- the best current technique is to use mixed precision with post-quantisation fine tuning to restore performance. That's most probably how all of the "turbo" models work. Take a model that was initially 16 or 32 bits per parameter during training, quantise it down to a mixture of 4, 8, and 16 bits, and then fix it up with an additional training pass that uses the original full-fat model's predictions as the loss function. With access to the raw parameters, it's possible to do this training such that all of the output weights are considered and adjusted during this phase instead of just the top word. Third parties fine-tuning against GPT4 chats can't do this, even with the collected samples, because they only have individual selected tokens/words instead of the full probability distribution.
I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
Anyone else seeing any evidence of this?
It's full of memes and people complaining its not as "good" as it was yesterday when it fails at completing their homework.
I would take anything said there with a big grain of salt, and stick to benchmarks.
It's like the AI equivalent of "the rest is left as an exercise to the reader" you'd find in old textbooks.
gpt4 should really be called DecartesGPT with this bs.
"I benchmarked on SAT reading, which is a nice human reference for reasoning ability. Took 3 sections (67 questions) from an official 2008-2009 test (2400 scale) and got the following results, here a SAT-like test:
- GPT3.5 - 690 (10 wrong) - GPT4 - 770 (3 wrong) - GPT4-turbo (one section at time) - 740 (5 wrong) - GPT4-turbo (3 sections at once, 9K tokens) - 730 (6 wrong)"
Source: https://twitter.com/wangzjeff/status/1721934560919994823?t=P...
You seem to be suggesting it got a bit worse, and the aider article seems to suggest gpt4 got a bit worse, although much faster at being a bit worse, while gpt3.5 got worse, then better, while faster.
However, in my opinion the first attempt score is more important, and Turbo does genuinely seem to lead there. There's still a possibility the updated training data has tainted the results.
Does it really make sense to play with this kind of power?
Better to be excited and learn the tool as it develops than to stick your head in the sand.
A world where AI has put literally everyone “out of a job,” meanwhile, is still so far from our current reality that IMO, it’s not worth making practical day-to-day decisions on unless you are directly involved in the development or regulation of AI.
2 years ago something like ChatGPT (as limited as it is) was “far from our current reality”.
I think it’s worthwhile to think ahead.
But two years ago, GPT-3 was already astounding. ChatGPT made some pretty huge gains, and GPT4 more still, but the thing that exists today was relatively predictable, compared to a world where human effort is no longer important.
Or do you think there will always be space for humans to provide value to other humans, even if machines surpass us in intelligence?
Even if AI can do everything, you'd still want the authentic human experience.
99% of people are far smarter than horses but people still pay to ride horses.
I don't see why someone wouldn't want to pay to... ride me... uh, in a matter of speaking, of course (of course). I mean, look at me.
How many working horses are there today vs before automobiles?
Heck, how many domestic horses are there today vs before automobiles?
Well, exactly. The less working horses there are, the more expensive and exclusive it would be to ride them.
There could be 1 trillion automobiles and I bet you, none of these automobiles would compare to riding a real, live horse.
Similarly, there could be 1 trillion AI robots, they could do everything better than a human, and yet I bet you'd still want to ride (or otherwise experience) a real, live human.
My point is that if automobiles were always better than horses in every way, then nobody would want horses. But even today, with the amazing automobiles that we have, some of which even faster and more reliable than most horses, it's clear that we still want horses.
My question is, if horses were as intelligent as us and they could have their basic needs met extremely cheaply, would they be willing to work at all, apart from the occasional ride? Because the horse labor pool would shrink immensely if they didn't really want to work.
So you expect there to be less of us?
Imagine everyone who is selling labor today out of a job.
> So you expect there to be less of us?
Less working people, not less people.
> Imagine everyone who is selling labor today out of a job.
If it's because they don't need it anymore, that's great!
What are the people not work going to eat?
We though AI would replace the low level jobs first, but it seems creative jobs are gone first (art, software developers, etc). Bear that in mind.
Frankly, if it's the higher end jobs getting replaced first that would likely spill over to the lower end ones as the those people who lost their jobs resort to taking lower end work to survive, flooding the market.
The shape of my career might change, but I doubt I'll be unable to find a job.
I think it's critical to be thinking about how to make sure wealth isn't simply funneled to a few capitalists who own everything simply by virtue of them being first, because it seems that's the future we're heading for if we aren't careful.
I think you, and I, and everyone on HN will be fine (more or less...) but I am worried about a wide swath of people who will get "left behind."
It would be pointless. Do you even know that 99% of horses can't speak English?
Anyway, my own experience tells me they're mostly just horsing around.
A lot more leisure and a lot less work than a few centuries ago...
Good times!
https://energyhistory.yale.edu/horse-and-mule-population-sta...
Clearly unsustainable, so I'm glad that problem solved itself. Phew! Can you imagine what it'd be like if they continued reproducing like a plague?
Anyway, the real question is: were they happier in 1910 when they were our slaves or are they happier now that they have more leisure?
How do you even measure happiness in a horse?
How do you measure happiness in humans?
Is happiness even real?
What is the meaning of life?
What does "42" mean?
In the past, new automation technologies often open up new possibilities in production capabilities in turn creating new jobs - specifically jobs that have not been automated yet.
AI though promises to be the universal automation, i.e. it can do any job. Thus even if new jobs show up, they will be taken over by AI too.
Then what?
> The shape of my career might change, but I doubt I'll be unable to find a job.
Question you should ask is why would anyone hire you when they can get AI to do the same job.
My guess is that some people will always be stubborn and prefer me over an AI giving them massages every now and then, even if I won't do them as perfectly.
But at that point in time, I think me giving one massage would perhaps be enough for me to retire for life, as long as I can abstain from buying luxury goods or services, i.e. those provided manually by other people (instead of AIs, whose labor would be practically free).
But why you? There are billions of other people who are out of work they could hire for the “authentic human experience”.
They already gave one massage, so now they can afford AI-made stuff for the rest of their lives, including AI massages for themselves.
AI labor is effectively free at that point, didn't you get the memo? :)
It’s not. Machines need power to run. Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
Did you forget about all the endless solar panels and wind turbines that the AI robots progressively built over the years, for an increasingly cheaper price? Energy is abundant now.
They also built a bunch of AI robot factories, so now we have effectively limitless AI robot labor. Everyone has AI robot assistants/friends now.
> Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
But they have something to give in return. These owners want authentic human experiences, which can only be provided by other people, not AI robots.
Authentic human experiences (or hand-built goods) are thus now much scarcer, since most people don't need to work anymore. They're sort of luxury goods/services. In this world, imagine how much you'd be willing to pay to have an exclusive, authentic house maid or hairdresser for one day, rather than just another AI robot, no matter how good it is.
Or imagine dining at a restaurant where every waiter and every cook are authentic humans, wow! An unforgettable (and prohibitively expensive) experience.
Of course, not everyone could afford that, unless they themselves are willing to provide such a service. Most people would eat at home (with food farmed, prepared/cooked and delivered by the AI robots and drones, of course), or just go to regular AI robot restaurants if they're feeling especially fancy.
There will also be an over supply of people willing to offer it because they have nothing else to offer in trade for basic necessities.
Rich people are notorious for seeking luxuries.
> There will also be an over supply of people willing to offer it because they have nothing else to offer in trade for basic necessities.
I don't know, perhaps you are right. I do hope you are wrong, though... :)
Also, I'm hoping if such an extreme situation would present itself, governments would be forced to step in and somehow ensure people would always have their basic needs met by, say, a strategic national AI army dedicated to provide those basic needs.
Speculating is fun, isn't it? lol
That would depend on who is your government.
An AI can't do any job that requires a body, so clearly that isn't true.
If you mean mental jobs, this could be true in the future, but right now, it's hard to say. You can imagine scenarios where an AI does 95% of a job but a human is still required for the last 5% (without which the first 95% is useless).
These tools are recent, but not THAT recent. GPT3 is nearly 4 years old, Copilot is 2 years old, ChatGPT is a year old, GPT4 is 8 months old. No large-scale employment changes resulting from AI are visible so far.
No reason they wouldn’t be provide physical bodies eventually.
This is the kind of scenario I was talking about in my initial comment. The naive assumption about this scenario is that it would result in 95% of jobs disappearing, but based on history it's more likely that we just do 2000% more stuff with the same amount of human effort.
And if AI really replaces 100% of human labor, are we really going to retain an economic system that only works when humans are doing labor?
Somehow that didn't happen, though.
Agriculture's share of the labor force went from something like 70% in the early 20th century to something like 5% now. You have a higher purpose than doing a robot's job badly -- or a farmer's. Find it.
Are there any other programming assistant packages that use the chatgpt api like this?
Regarding rate limits, it might be an idea to have configurable delays built in to the testing code to prevent hitting limits.