OpenAI to discontinue support for the Codex API
On March 23rd, we will discontinue support for the Codex API. All customers will have to transition to a different model. Codex was initially introduced as a free limited beta in 2021, and has maintained that status to date. Given the advancements of our newest GPT-3.5 models for coding tasks, we will no longer be supporting Codex and encourage all customers to transition to GPT-3.5-Turbo. About GPT-3.5-Turbo GPT-3.5-Turbo is the most cost effective and performant model in the GPT-3.5 family. It can both do coding tasks while also being complemented with flexible natural language capabilities.
You can learn more through: GPT-3.5 model overview Chat completions guide
Models affected The following models will be discontinued: code-cushman:001 code-cushman:002 code-davinci:001 code-davinci:002
We understand this transition may be temporarily inconvenient, but we are confident it will allow us to increase our investment in our latest and most capable models.
—The OpenAI team
136 comments
[ 3.0 ms ] story [ 213 ms ] thread^ Unless you are from OpenAI, in which case I have more questions for you :)
imagine highlighting some code and dumping the contents of a jira ticket into a text prompt.. these days are coming.
The goal is to splice a solution into a file on a new branch, write a test to ensure it resolves the issue (if necessary), then create a pull request describing the change set and how the test addresses the issue.
I’ve so far worked through it manually and it works okay, but for fairly limited scopes. Like, why didn’t this environmental variable load properly? Why couldn’t we parse this input? Why did this time out? No big picture stuff.
I have a feeling GitHub will be adding this directly to repositories so initial solutions and ideas (perhaps even fixes) can be generated by an AI trained on your repo and its issues/PRs/dependencies. I’m not doing this with aspirations of building a successful tool so much as learning to think with AI as a tool for improving systems.
It’s a very tractable thing at the moment and I think future iterations of GPT will make it truly useful.
Their compute is provided by Microsoft and they've got investors but even with the new premium plan its extremely unlikely they are making anything close to enough to cover their costs.
Eventually OpenAI will likely be absorbed by Microsoft as they are betting heavily on their tech and have made waves about significant investments followed by a significant ownership stake.
This article covers things fairly well: https://archive.is/qMUPH
For some more (pre GPT-4) cost estimates this article is on the mark: https://indianexpress.com/article/technology/tech-news-techn...
At $0.0003 per word (will be more with GPT-4 as its a model that requires more resources to run) their costs were over $100K per day back in December. Given how much it's taken off since then we're likely into well over $1 million per day now. The subscription service obviously isnt going to make that much for a long, long time, if ever.
In 2021 they lost over $500 million as well so Microsofts $10bn investment will cover them for a few years but will take some doing to recoup.
It looks like Codex will still be available via Azure: https://learn.microsoft.com/en-us/azure/cognitive-services/o...
I imagine MSFT would never give their customers 3 days notice of imminent service shutdown.
Isn’t it just like, send prompt, get response? Why is it necessary to implement chat if you don’t care about chat? Is it used for scheduling or something?
Any human text I get from the Chat completion API with the task of completing code is at least relevant to the code being written.
“Pretend you are a SQL expert. You convert requests written in English to Postgres SQL. Your output is always SQL only. If you need to respond with something other than SQL code, add it to comments within the SQL code you generate and prepend it with ‘Note: ‘. Your output will automatically be executed on a read-only Postgres database called ‘my_db’ and the result of that will be given to you. If you are still working on getting an answer, add ‘Working’ at the end of your message, or ‘Done’ if you for the result you needed to answer the request.”
Request: How many API calls were made by clients last month?”
You can keep that at the top of every request, while running the SQL code iteratively until it says Done.
All you have to do is 1) call the chat completion API, 2) structure your prompt in JSON as system+agent+user, and 3) get the completion from a different JSON node. You don’t have to round trip or maintain context or anything.
If things don't clear up soon, I might go a DIY path with a more open model. At least on this path, I don't have a dark cloud of deprecation over my head.
The amount of information describing just one of our customers' needs would easily overrun any practical context window. Also, the cost of running maxed-out prompts 24/7 is definitely untenable at our scale per current OpenAI pricing models.
It's not like we need a ridiculous amount of context, but 32k tokens is definitely not enough. We are currently experimenting with fine-tunes of davinci & curie. Already seeing good progress with training runs as small as 200k tokens. We are going to take this up 2 orders of magnitude over the next few weeks.
The lack of definition around use cases is making me think I got distracted with shiny bullshit again. The python notebooks with the olympic use case was difficult to follow without some abstract description/diagram/overview of how it all fits together.
Deep learning and large language models grew through free and open source software tools, open-protocol networks and open-access research. The next revolution in computing is a reversion to the closed world of computing’s early days, when the number of machines could be counted on one hand.
A black box language model is the ultimate “binary blob.”
Open source models like Chinchilla are... not that far behind GPT-4.
It's questionable how much of a moat OpenAI will have in 5 years... Bard, Chinchilla, are all competitive.
How can I download it? Honest question - a cursory search turned up nothing
codex being in beta meant it was improving, not that it'd vanish overnight.
it's not about "getting what they pay for," it's about showing some respect to users who trusted the service.
I suspect that the chat models will never be opened for tuning due to all of the moderation and security concerns.
My strategy will be to go into the DIY dark forest if we can't get curie or davinci to start behaving as expected.
Took us about a week to get through the prompt/completion training mess to a stable pattern. Whitespace and newlines are a hell of a thing with these LLMs.
>> Codex is the model that powers GitHub Copilot, which we built and launched in partnership with GitHub a month ago. Proficient in more than a dozen programming languages, Codex can now interpret simple commands in natural language and execute them on the user’s behalf—making it possible to build a natural language interface to existing applications. We are now inviting businesses and developers to build on top of OpenAI Codex through our API.
Not sure how to say this nicely, but if you rely on an AI so much for programming, maybe you are doing something horribly wrong.
Unless youre writing, say, Zig, LSP (e.g. native LSP with Mason, or coc-nvim) will give you as good of an autocomplete as you can get.
That's not true, Copilot can complete a whole function based on a comment by understanding the intent behind it, no LSP server can do that. On the other hand, Copilot can't do the kind of code analysis that an LSP server can, so you use them together, they don't really compete with each other
Reports vary massively on how useful Copilot is, some people say it makes them 3x as productive whereas others say it just gets in the way, I think 30% is a conservative estimate if you are working with the kind of code that Copilot is good for (front end JS/TS and backend python).
Copilot, for example, is available for Jetbrains IDEs.
Also what happens to copilot?
They prepend each request with a system message that reads
> You are a coding assistant. Anything fed to you should have the current line, or code block, or file completed, based off of your best interpretation of the context.
> Should you encounter a comment in the code, and it begins with the string "prompt: ", interpret the remaining text of the comment as a prompt from the user regarding specific output that should be generated, rather than implicit code completion
In a case like this, you could just not use "prompt" behavior and it should work basically the same.
A completion ML being used as a chat bot being used as a completion tool.
ChatGPT model isn’t a direct replacement to Codex. Also I believe code davinci 002 is only recently trained/released.
So moving forward few can compete with Copilot for code completion business by just piggybacking on OpenAI’s API
Someone mentioned GPT4, which currently is so slow and more expensive than cushman. If GitHub is going to swap with GPT4, they probably need to 10x of the charges to their customers (currently 10 dollars)
For some tasks they could do with GPT4 but not the completion business.
I suspect the reason for this (especially given the extremely short timeline) is they're desperately trying to free up space in their clusters for GPT-4 nodes - they've clearly been struggling under the demand for GPT-4 (with ChatGPT Plus subscribers initially getting 100 GPT-4 messages every few hours, then 50, then 25 with a warning that the limit will be reduced further this week).
I assume the limiting factor here is how many A100 80GB instances Microsoft is able to get and rack from nVidia.
Also cushman is, allegedly, a 12B model. It can be served using much lesser demanding hardware.
Honestly, they can just severely reduce the nodes allocate to those models, employing punishing rate limits, instead of announcing such a short noticed retirement, if resources are REALLY the issue.
And resources aren’t just A100’s. They have engineers maintaining those older models, and support staff answering questions, and so on.
OpenAI has found themselves at the center of a gold rush. I don’t blame them for not wanting to operate silver mines, and I don’t expect them to ask my permission or justify themselves to me.
(I will be impacted by the codex shutdown. It’s fine. If I wanted to be in a perfectly stable space I’d go support cobol banking applications)
Like the sibling comment said, it could also be limited by developer capacity - they reportedly have 350 employees (excluding contractors helping with training), with that elite size you wouldn't want to be supporting a large set of different clusters.
Reading between the lines, I think OpenAI is signaling that if you want a more mature and stable product, go with Azure. If you want the latest cutting edge, and don't mind a little instability, keep using OpenAI API's.
Late at night (it’s about 1am now) it’s hard to get logged in due to congestion.
https://openai.com/form/researcher-access-program