Did they actually re-release 0125 as a retrained model with newer data, or was that an oversight? The date of Dec 2023 seems to suggest that was always the training time of the snapshot, rather than OpenAI silently re-revving the 0125 model with this doc update being their announcement. But it's a preview model, so maybe anything goes.
In my experience, these updates don't matter that much. For example, GPT-4 still uses Pydantic v1 syntax (which for the most part is deprecated) even though Pydantic v2 came in June 2023. the information regarding v1 is so much more in GPT-4's training data that "on average" it picks that instead of the new v2 syntax data.
This has been a major pain point for me, because GPT-4 constantly uses @validator, "const", "always", etc. features that don't work on v2.
This is why with future models and their effectively unlimited context size you'll simply concatenate the entire API codebase you're using into the context of your prompt.
I've done similar already, but ChatGPT still tends to rewrite the code to the old API, even if it's correctly using the new API. Ironically, I just had this happen to me while using ChatGPT 4 to use the OpenAI API.
Although the attention is really good at focusing on specific information, I've had problems with this approach. It seems like if I ask a task and provide a large enough API it gets distracted or ignores the chunk of text and does its own thing anyway.
Also it's still more expensive to increase the context length
have you tried adding a clause that all output must come in pydantic v2 in your prompt? additionally you can add a few common fewshot examples to get it in the pydantic v2 state
I tried exporting the Pydantic documentation page on syntax changes as PDF, feeding that to GPT-4, and then telling it to be careful about Pydantic syntax changes. That helped it use the new syntax, but we can't feed in the entire Pydantic doc so it still didn't get the details right.
Probably not right now, the standard process would involve re-running a 'finetuning' part after any update to the underlying model, and while that's far less expensive than the main training, it's probably not something you'd want to do every day.
I'm asking you this since you sound like you might know. At what point in the process do they add in the guardrails/"baby-proofing"? And how do they do it?
There's usually a two or three step training procedure, first training to predict the next word on a huge corpus of text (billions or trillions of words), then possibly some instruction tuning (giving the model question & answer pairs and training on the answer) and then finally RLHF (or RLAIF, DPO etc) where the model is trained to match human preferences. It's this last step that is used to increase the helpfulness & harmlessness of the model, training it to not respond to certain topics.
In general, the core language model is simply trained on a very large amount of unannotated text (which is the most time-consuming and expensive part), but a language model is not directly very useful in the role of e.g. a chat agent, it quite literally tries to continue text and that sometimes is what you want and sometimes isn't.
The second step is fine-tuning the model on a much smaller set of annotated data which specify that it should actually "do something" in its responses and what it should do; it "teaches" it that it should actually answer the questions instead of e.g. continuing on with a list of more questions in the same vein, and most such training sets also "teach" it that for certain questions the appropriate response is a refusal.
If you have the original core model (before that instruction tuning) then you can repeat the same process but instead replace the instruction training set with a different one, so you can "instruct" the model to behave differently. Here is a nice and informative article from Eric Hartford about how he did that to make certain 'uncensored' models - https://erichartford.com/uncensored-models
Why do you assume that Gemini doesn't have these knowledge cut-offs?
If I ask Gemini, for example "Give me three important events that happened during May 13, year 2023.", then it says that this is "in the future" and responds that it can provide some guesses "Based on publicly available information from early February 2023", so that probably is the cut-off date for the model.
However, I would assume that (just like Bing) for certain questions it can pull in extra information from web searches - the model can be old, but if the system puts some retrieved document(s) in the prompt context so that the model can use them for generating the response, it can use that (limited) fresh information as well.
I might have been a stupid assumtion indeed but I assumed it based on the fact that Gemini does not respond with the same shit chatGPT says regarding its knowledge cut-off.
Not entirely related to just GPT-4, but I feel like not many updates will matter for ChatGPT until they completely overhaul the system prompt. This seems to be the reason why ChatGPT has gotten "dumber". https://twitter.com/AlphaSignalAI/status/1757466498287722783
How are they filtering out posts from existing generative ai models in the training set? Or do they just not bother and train on increasingly polluted datasets?
We will learn whether the degeneration of generative AI leading to model collapse as predicted by some is inevitable (“content is generated as an ‘average’ from mathematical inputs, degenerating because you cannot jump outside the inputs).
It's weird, I use the standard "gpt-4" through the API and it's still stuck on September 2021. Annoying. According to the docs it's supposed to point to the latest version but it's never updated.
Ps I also tried gpt-4-turbo but the API rejects it for some reason. Same with the "numbered" ones listed in the documentation, I only seem to have access to the standard gpt-4 somehow.
47 comments
[ 4.6 ms ] story [ 143 ms ] threadSee this 25 day old screenshot of the docs which says knowledge cutoff April 2023 for the 0125 model
https://www.reddit.com/r/ChatGPT/comments/19fhb6h/new_gpt_4_...
This help article which was updated "this week" also still mentions April 2023
https://help.openai.com/en/articles/8555510-gpt-4-turbo-in-t...
And in the announcement for 0125 they didn't say anything about the knowledge cutoff change
https://openai.com/blog/new-embedding-models-and-api-updates
1) Replaced a pinned version with a new model (problematic for response consistency), or,
2) Decoupled knowledge cutoff from the model (how!?)
> when were you last updated?
My last update was in April 2023.
This has been a major pain point for me, because GPT-4 constantly uses @validator, "const", "always", etc. features that don't work on v2.
Also it's still more expensive to increase the context length
Both models were not able to have any pydantic v2 training data.
The second step is fine-tuning the model on a much smaller set of annotated data which specify that it should actually "do something" in its responses and what it should do; it "teaches" it that it should actually answer the questions instead of e.g. continuing on with a list of more questions in the same vein, and most such training sets also "teach" it that for certain questions the appropriate response is a refusal.
If you have the original core model (before that instruction tuning) then you can repeat the same process but instead replace the instruction training set with a different one, so you can "instruct" the model to behave differently. Here is a nice and informative article from Eric Hartford about how he did that to make certain 'uncensored' models - https://erichartford.com/uncensored-models
If I ask Gemini, for example "Give me three important events that happened during May 13, year 2023.", then it says that this is "in the future" and responds that it can provide some guesses "Based on publicly available information from early February 2023", so that probably is the cut-off date for the model.
However, I would assume that (just like Bing) for certain questions it can pull in extra information from web searches - the model can be old, but if the system puts some retrieved document(s) in the prompt context so that the model can use them for generating the response, it can use that (limited) fresh information as well.
https://vue.ai/blog/ai-transformation/the-degeneration-of-ge...
We have truly hit the limit of names in this industry.
Ps I also tried gpt-4-turbo but the API rejects it for some reason. Same with the "numbered" ones listed in the documentation, I only seem to have access to the standard gpt-4 somehow.