With how good gpt-3.5-turbo-0613 is (particularly with system prompt engineering), there's no longer as much of a need to use the GPT-4 API especially given its massive 20x-30x price increase.
The mass adoption of the ChatGPT APIs compared to the old Completion APIs proves my initial blog post on the ChatGPT API correct: developers will immediately switch for a massive price reduction if quality is the same (or better!): https://news.ycombinator.com/item?id=35110998
I have a startup of legal AI, the quality jump from GPT3.5 to GPT4 in this domain is straight mind-blowing, GPT3.5 in comparison is useless. But I see how in more conversational settings GPT3.5 can provide more appealing performance/price.
I suggested to my wife that ChatGPT would help with her job and she has found ChatGPT4 to be the same or worse as ChatGPT3.5. It’s really interesting just how variable the quality can be given your particular line of work.
Legal writing is ideal training data: mostly formulaic, based on conventions and rules, well-formed and highly vetted, with much of the best in the public domain.
Medical writing is the opposite, with unstated premises, semi-random associations, and rarely a meaningful sentence.
> Legal writing is ideal training data: mostly formulaic, based on conventions and rules, well-formed and highly vetted, with much of the best in the public domain.
That makes sense. The labor impact research suggests that law will be a domain hit almost as hard as education by language models. Almost nothing happens in court that hasn't occured hundreds of thousands of times before. A model with GPT-4 power specifically trained for legal matters and fine tuned by jurisdiction could replace everyone in a courtroom. Well there's still the bailiff, I think that's about 18 months behind.
I am building an extensive LLM-powered app, and had a chance to compare the two using the API. Empirically, I have found 3.5 to be fairly unusable for the app's use case. How are you evaluating the two models?
It depends on the domain, but chain of thought can get 3.5 to be extremely reliable, and especially with the new 16k variant
I built notionsmith.ai on 3.5: for some time I experimented with GPT 4 but the result was significantly worse to use because of how slow it became, going from ~15 seconds per generated output to a minute plus.
And you could work around that with things like streaming output for some use cases, but that doesn't work for chain of thought. GPT 4 can do some tasks without chain of thought that 3.5 required it for, but there are still many times where it improves the result from 4 dramatically.
For example, I leverage chain of thought in replies to the user when they're in a chat and that results in a much better user experience: It's very difficult to run into the default 'As a large language model' disclaimer regardless of how deeply you probe a generated experience when using it. GPT 4 requires the same chain of thought process to avoid that, but ends up needing several seconds per response, as opposed to 3.5 which is near-instant.
-
I suspect a lot of people are building things on 4 but would get better quality of output if they used more aspects of chain of thought and either settled for a slower output or moved to 3.5 (or a mix of 3.5 and 4)
It depends a lot on the domain, even for CoT. I don't think there are enough NLU evaluations just yet to robustly compare GPT-3.5 w/ CoT/SC vs. GPT-4 wrt domain.
For instance, with MATH dataset, my own n=500 evaluation showed no difference between GPT-3.5 (w/ and w/o CoT) and GPT-4. I was pretty surprised by that.
My experience is that GPT-3.5 is not better or even nearly as good as GPT-4. Will it work for most use cases? Probably, yes. But GPT-3.5 effectively ignores instructions much more often than GPT-4 and I've found it far far easier to trip up with things as simple as trailing spaces; it will sometimes exhibit really odd behavior like spelling out individual letters when you give it large amounts of text with missing grammar/punctuation to rewrite. Doesn't seem to matter how I setup the system prompt. I've yet to see GPT-4 do truly strange things like that.
The initial gpt-3.5-turbo was flakey and required significant prompt engineering. The updated gpt-3.5-turbo-0613 fixed all the issues I had even after stripping out the prompt engineering.
It's definitely gotten better, but yeah, it really doesn't reliably support what I'm currently working on.
My project takes transcripts from YouTube, which don't have punctuation, splits them up into chunks, and passes each chunk to GPT-4 telling it to add punctuation with paragraphs. Part of the instructions includes telling the model that, if the final sentence of the chunk appears incomplete, to just try to complete it. Anyway, GPT-3.5-turbo works okay for several chunks but almost invariably hits a case where it either writes a bunch of nonsense or spells out the individual letters of words. I'm sure that there's a programmatic way I can work around this issue, but GPT-4 performs the same job flawlessly.
If GPT 4 is working for you I wouldn't necessarily bother with this, but this is a great example of where you can sometimes take advantage of how much cheaper 3.5 is to burn some tokens and get a better output. For example I'd try asking it for something like :
{
"isIncomplete": [true if the chunk seems incomplete]
"completion": [the additional text to add to the end, or undefined otherwise]
"finalOutputWithCompletion": [punctuated text with completion if isIncomplete==true]
}
Technically you're burning a ton of tokens having it state the completion twice, but GPT 3.5 is fast/cheap enough that it doesn't matter as long as 'finalOutputWithCompletion' is good. You can probably add some extra fields to get an even nicer output than 4 would allow cost-wise and time-wise by expanding that JSON object with extra information that you'd ideally input like tone/subject.
I've done exactly this for another project. I'd recommend grabbing an open source model and fine-tuning on some augmented data in your domain. For example: I grabbed tech blog posts, turned each post into a collection of phonemes, reconstructed the phonemes into words, added filler words, and removed punctuation+capitalization.
Sounds interesting, any chance you could share either your end result that you used to then fine-tune with, or even better the exact steps (ie technically how you did each step you already mentioned)?
And what open LLM you used it with / how successful you've found it?
I use it to generate nonsense fairytales for my sleep podcast (https://deepdreams.stavros.io/), and it will ignore my (pretty specific) instructions and add scene titles to things, and write the text in dramatic format instead of prose, no matter how much I try.
I mostly use it for generating tests, making documentation, refactoring, code snippets, etc. I use it daily for work along with copilot/x.
In my experience GPT3.5turbo is... rather dumb in comparison. It makes a comment explaining what a method is going to do and what arguments it will have - then misses arguments altogether. It feels like it has poor memory (and we're talking relatively short code snippets, nothing remotely near it's context length).
And I don't mean small mistakes - I mean it will say it will do something with several steps, then just miss entire steps.
GPT3.5turbo is reliably unreliable for me, requiring large changes and constant "rerolls".
GPT3.5turbo also has difficulty following the "style/template" from both the prompt and it's own response. It'll be consistent then just - change. An example being how it uses bullet points in documentation.
Codex is generally better - but noticeably worse then GPT4 - it's decent as a "smart autocomplete" though. Not crazy useful for documentation.
Meanwhile GPT4 generally nails the results, occasionally needing a few tweaks, generally only with long/complex code/prompts.
tl;dr - In my experience for code GPT3.5turbo isn't even worth the time it takes to get a good result/fix the result. Codex can do some decent things. I just use GPT4 for anything more then autocomplete - it's so much more consistent.
If you're manually interacting with the model, GPT 4 is almost always going to be better.
Where 3.5 excels is with programmatic access. You can ask it for 2x as much text between setup so the end result is well formed and still get a reply that's cheaper and faster than 4 (for example, ask 3.5 for a response, then ask it to format that response)
I think this is very very use-case dependent, and your use case != everyone's use case.
In my experience, GPT-4 is night and day better than 3.5 turbo for almost everything I use OpenAI for.
> "We envision a future where chat-based models can support any use case. Today we’re announcing a deprecation plan for older models of the Completions API"
nooooo they are deprecating the remnants of the base models
They're deprecating all the completion/edit models.
The chat models constantly argue with you on certain tasks and are highly opinionated. A completion API was a lot more flexible and "vanilla" about a wide variety of tasks, you could start a thought, or a task, and truly have it complete it.
The chat API doesn't complete, it responds (I mean of course internally it completes, but completes a response, rather than a continuation).
I find this a big step back, I hope the competition steps in to fill the gaps OpenAI keeps opening.
not in the article: is plugin usage available to paying customers everywhere now? i still can't see the ui for it. im in canada and use pro. internet says it was out for everyone in may..
With this is the death of any uncensored usage of their models. Davinci 3 is the most powerful model where you can generate any content by instructing it via the completions API - chat GPT 3 models will not obey requests for censored or adult content.
A big enough hole presents a wedge for new entrants to get started.
OpenAI will never fulfill the entire market, and their moat is in danger with every other company that has LLM cash flow.
They want to become the AWS of AI, but it's becoming clear they'll lose generative multimedia. They may see the LLM space become a race to the bottom as well.
Let's hope so - the amount of control they have over this is a great evil. Many of us have experienced the potential of a less moderated GPT4, and we all know that somewhere out there, they have the full unmoderated version. What are they using it for? What powers have got their hands on this thing?
Has anyone been able to come up with a way to keep track of GPT-4 performance over time? I'm told that the API is explicit about changes to models and that the Chat interface is not.
API call responsiveness to the GPT-4 model varies hugely throughout the day. The #1 datapoint in measured responsiveness is slowdown associated with lunch-time use as noon sweeps around the globe.
Thank you for the response, I should have been clearer. I meant performance as an LLM. Essentially, I am concerned that they are quietly nerfing the tool. The Chat interface is now very verbose and constantly warning me about "we should always do this and that" which is bloody exasperating when I'm just trying to get things done.
I made up an example here to illustrate, but it's just very annoying because sometimes it puts at the beginning, slowing down my interaction, and it now refuses to obey my prompts to leave caveats out.
yeah, its annoying and you have to foot the bill for it.
looking at your sample and using character count as a rough proxy for tokens, (465/(1581-465))*100 means they added ~42% token count cost to your response explicitly adding caveats which you dont want. fun!
Hmm, when I try to change model name to "gpt-4" I get the "The model: `gpt-4` does not exist" error message. We are an API developer with a history of successful payments.. is there anything we need to do on our side to enable this, anyone know?
Outside of the headline, there is some major stuff hiding in here:
- new gpt-3.5-turbo-instruct model expected "in the coming weeks"
- fine tuning of 3.5 and 4 expected this year
I am especially interested in gpt-3.5-turbo-instruct, as I think that the hype surrounding ChatGPT and "conversational LLMs" has sucked a lot of air out of what is possible with general instruct models. Being able to fine tune it will be phenomenal as well.
is there any ETA on when the knowledge cutoff date will be improved from September, 2021?
I do not really understand the efforts that went on behind the scenes to train GPT models on factual data. Did humans have to hand approve/decline responses to increase its score?
"America is 49 states" - decline
"America is 50 states" - approve
Is this how it worked at a simple overview? Do we know if they are working on adding the rest of 2021, then 2022, and eventually 2023? I know it can crawl the web with the Bing addon but, it's not the same.
I asked it about Maya Kowalski the other day. Sure it can condense a blog post or two, but it's not the same as having the intricacies as if it actually was trained/knew about the topic.
In all my GPT-4 API (python) experiments, it takes 15-20 seconds to get a full response from server, which basically kills every idea I've tried hacking up because it just runs so slowly.
Has anyone fared better? I might be doing something wrong but I can't see what that could possibly be.
Streaming. If you’re expecting structured data as a response, request YAML or JSONL so you can progressively parse it. Time to first byte can be milliseconds instead of 15-20s. Obviously, this technique can only work for certain things, but I found that it was possible for everything I tried.
We use it to generate automatic insights from survey data at a weekly cadence for Zigpoll (https://www.zigpoll.com). This makes getting an instant response unnecessary but still provides a lot of value to our customers.
GPT-4 fine tuning capability will be huge. It may end up just making fine tuning OSS LLMs pointless, esp if they keep lowering GPT-4 costs like they have been.
I know everyone's on text-embedding-ada-002, so these particular embedding deprecations don't really matter, but I feel like if I were using embeddings at scale, the possibility that I would one day lose access to my embedding model would terrify me. You'd have to pay to re-embed your entire knowledge base.
> We recognize this is a significant change for developers using those older models. Winding down these models is not a decision we are making lightly. We will cover the financial cost of users re-embedding content with these new models. We will be in touch with impacted users over the coming days.
I don't have any example responses at hand here. But this was a prompt (that had a shitty pre-prompt of conversational messages) running on davinci-003.
Can you try notionsmith.ai and let me know what you think?
I've been working on LLMs for creative tasks and believe a mix of chain of thought and injecting stochasticity (like instructing the LLM to use certain random letters pulled from an RNG in a certain way at certain points) can go a long way in terms of getting closer to human-like creativity
Biggest news here from a capabilities POV is actually the gpt-3.5-turbo-instruct model.
gpt-3.5-turbo is the model behind ChatGPT. It's chat-fine-tuned which makes it very hard to use for use-cases where you really just want it to obey/complete without any "chatty" verbiage.
The "davinci-003" model was the last instruction tuned model, but is 10x more expensive than gpt-3.5-turbo, so it makes economical sense to hack gpt-3.5-turbo to your use case even if it is hugely wasteful from a tokens point of view.
One is tuned for chat. It has that annoying ChatGPT personality. Instruct is a little "lower level" but more powerful. It doesn't have the personality. It just obeys. But it is less structured, there are no messages from user to AI, it is just a single input prompt and a single output completion.
the existing 3.5turbo is what you would call a "chat" model.
The difference between them is that the chat models are much more... chatty - they're trained to act like they're in a conversation with you. The chat models generally say things "Sure, I can do that for you!", and "No problem! Here is". The conversation style is generally more inconsistent in it's style. It can be difficult to make it only return the result you want, and occasionally it'll keep talking anyway. It'll also talk in first person more, and a few things like that.
So if you're using it as an API for things like summarization, extracting the subject of a sentence, code editing, etc, then the chat model can be super annoying to work with.
I'm interested in the cost of gpt-3.5-turbo-instruct. I've got a basic website using text-davinci-003 that I would like to launch but can't because text-davinci-003 is too expensive. I've tried using just gpt-3.5-turbo but it won't work because I'm expecting a formatted JSON to be returned and I can just never get consistency.
I'm assuming they will price it the same as normal gpt-3.5-turbo. I won't use it if it's more than 2x the price of turbo, because I can usually get turbo to do what I want, it just takes more tokens sometimes.
Have you tried getting your formatted JSON out via the new Functions API? I does cure a lot of the deficiencies in 3.5-turbo.
You need to use the new OpenAI Functions API. It is absolutely bonkers at returning formatted results. I can get it to return a perfectly formatted query-graph a few levels deep.
With the latest 3.5-turbo, you can try forcing it to call your function with a well-defined schema for arguments. If the structure is not overly complex, this should work.
i’ve had it come up with new function names, or prepend some prefix to the names of functions. i had to put some cleverness in on my end to run whatever function was close enough.
You can try to force JSON output using function calling (you have to use either the gpt-3.5-turbo-0613 or gpt-4-0613 model for now).
Think of the properties you want in the JSON object, then send those to ChatGPT as required parameters for a function (even if that function doesn't exist).
# Definition of our local function(s).
# This is effectively telling ChatGPT what we're going to use its JSON output for.
# Send this alongside the "model" and "messages" properties in the API request.
functions = [
{
"name": "write_post",
"description": "Shows the title and summary of some text.",
"parameters": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "Title of the text output."
},
"summary": {
"type": "string",
"description": "Summary of the text output."
}
}
}
}
]
I've found it's not perfect but still pretty reliable – good enough for me combined with error handling.
I'm hoping gpt-3.5-turbo-instruct isn't super neutered like chatgpt. davinci-003 can be a lot more fun and answer on a wide range of topics where ChatGPT will refuse to answer.
no expert, but from my messing around I gather the chat models are tuned for conversation, for example, if you just say 'Hi', it will spit out some 'witty' reply and invite you to respond, it's creative with it's responses. On the other hand, if you say 'Hi' to an instruct model, it might say something like, I need more information to complete the task. Instruct models are looking for something like 'Write me a twitter bot to make millions'... in this case, if you ask the same thing again, you are somewhat more like to get the same, or similar result, this does not appear so true with a chat model, perhaps a real expert could chime in :)
I imagine the API quality isnt nerfed on a given day like ChatGPT can be.
There was no question something happened in January with ChatGPT, weirdly would refuse to answer questions that were harmless but difficult(Give me a daily schedule of a stoic hedonist)
Every once in a while, I see redditors complain of it being nerfed.
Sometimes I go back to gpt3.5 and am mind boggled how much worse it is.
Makes me wonder if they keep increasing the version number while dumbing down the previous model.
With an API, being unreliable would be a deal-breaker. Looking forward to people fine-tuning LLMs with GPT4 API. I'd love it for medical purposes, I'm so worried of a future where the US medical cartels ban ChatGPT for medical purposes. At least with local models, we don't have to worry about regression.
The capability of the latest model will be like a Shepard tone: always increasing, never improving. Meanwhile their internal version will be 100x better with no filtering.
I feel like it's code generation abilities have also been nerfed. In the past I got almost excellent code from GPT-4, somehow these days I need multiple prompts to get the code I want from GPT-4.
Would gladly pay more for a none nerfed version if they were actually honest.
The current versions is close to the original 3.5 version, while 3.5 has become horribly bad, such a scam to not disclose what's going on, especially for a paid service.
Oh, it's not too hard to see how the spend that Microsoft put into building the data centers where GPT-4 was trained attracted national security interest even before it went public. The fact that they were even allowed to release it publicly is likely due to its strategic deterrence effect and that they believed the released version was already a dumbed-down version.
The fact that rumors about GPT-5 were quickly suppressed and the models were dumbed down even more cannot be entirely explained by excessive demand. I think it's more likely that GPT-3.5 and GPT-4 demonstrated unexpected capabilities in the hands of the public leading to a pull back. Moreover, Sam Altman's behaviors changed dramatically between the initial release and a few weeks afterward -- the extreme optimism of a CEO followed by a more subdued, even cowed, demeanor despite strong enthusiasm from end-users.
OpenAI cannot do anything without Microsoft's data center resources, and Microsoft is a critical defense contractor.
Anyway, personally, I'm with the crowd that thinks we're about to see a Cambrian explosion of domain-specific expert AIs. I suspect that OpenAI/Microsoft/Gov is still trying to figure out how much to nerf the capability of GPT-3.5 to tutor smaller models (see "Textbooks are all you need") and that's why the API is trash.
I was playing with the API and found that it returned better answers than ChatGPT. ChatGPT isn't even able to solve simple Python problems anymore, even if you try to help it. And some time ago it did these same problems with ease.
My guess is that they began to restrict ChatGPT because they can't sell that. They probably want to sell you CodeGPT or other products in the future so why would they give that away for free? ChatGPT is just a teaser.
Probably some combination of quantizing down from original fp16 weights and changes to the system prompt used for chat. Both can cause degraded quality, the former more than the latter.
"ChatGPT isn't even able to solve simple Python problems anymore, even if you try to help it. And some time ago it did these same problems with ease."
This is my experience also. I have not formally benchmarked the different releases, but specifically for Python coding ChatGPT 4 got considerably worse with the latest updates.
I agree. It is difficult to say what happened exactly but I am certain that I got all the answers and very few canned responses. Whatever they did for safety has degraded the product.
I keep reading “GPT4 got nerfed” but I have been using from day 1, and while it definitely gives bad answers, I cannot say that it was nerfed for sure.
Is there any actual evidences other than some user subjective experiences?
ChatGPT doesn't use the API. It uses the same underlying model with a bunch of added prompts (and possibly additional fine-tuning?) to add to make it conversational.
One would pay because what they get out of chatGPT provides value, of course. Keep in mind that the users of these 2 products can be (and in fact are) different — chatGPT is a lot friendlier (from a UX perspective) than using the API playground (or using the API itself).
I think the clearest evidence is Microsofts paper where they show abilities at various stages during training[1]... But in a talk [2], they give more details... The unicorn gets worse during the finetuning process.
Noobie follow up question: Should we put any trust into “Sparks of intelligence” I thought it was regarded as a Microsoft marketing piece, not a serious paper.
See my comment elsewhere on this post. Greg Brockman, head of strategic initiatives at OpenAI, was talking at a round table discussion in Korea a few weeks ago about how they had to start using the quantized (smaller, cheaper) model earlier in 2023. I noticed a switch in March 2023, with GPT-4 performance being severely degraded after that for both English-language tasks as well as code-related tasks (reading and writing).
Curious as to whether theres a more general rule at play there about filtering interfering with getting good answers. If there is that's a scary thought from an ethics perspective.
I hit rate limits and “model is busy with other requests” frequently while just developing a highly concurrent agent app. Especially with the dated (e.g. -0613) or now -16k models.
Instead of the model changing, it’s equally likely that this is a cognitive illusion. A new model is initially mind-blowing and enjoys a halo effect. Over time, this fades and we become frustrated with the limitations that were there all along.
Check out this post from a round table dialogue with Greg Brockman from OpenAI. The GPT models that were in existence / in use in early 2023 were not the performance-degraded quantized versions that are in production now: https://www.reddit.com/r/mlscaling/comments/146rgq2/chatgpt_...
No it's definitely changed a lot. The speedups have been massive (GPT 4 runs faster now than 3.5-turbo did at launch) and they can't be explained with just them rolling out H100s since that's just a 2x inference boost. Some unknown in-house optimization method aside, they've probably quantized the models down to a few bits of precision which increases perplexity quite a bit. They've also continued to RHLF tune to make them more in-line with their guidelines and that process has been shown to decrease overall performance before GPT 4 even launched.
But given the rumored architecture (MoE) it would make complete sense for them to dynamically scale down the number of models used in the mixture during periods of peak load.
I recently completed some benchmarks for code editing that compared the Feb (0301) and June (0613) versions of GPT-3.5 and GPT-4. I found indications that the June version of GPT-3.5 is worse than the Feb version.
After reading, I don't think <5% points is helpful to add to discussion here without pointing it out explicitly, people are asserting much wilder claims, regularly
I haven't come across any other systematic, quantitative benchmarking of the OpenAI models' performance over time, so I thought I would share my results. I think my results might argue that there has been some degradation, but not nearly the amount that you often hear people's annecdata about.
But unfortunately, you have to read a ways into the doc and understand a lot of details about the benchmark. Here's a direct link and excerpt of the relevant portion:
The benchmark results have me fairly convinced that the new gpt-3.5-turbo-0613 and gpt-3.5-16k-0613 models are a bit worse at code editing than the older gpt-3.5-turbo-0301 model.
This is visible in the “first attempt” portion of each result, before GPT gets a second chance to edit the code. Look at the horizontal white line in the middle of the first three blue bars. Performance with the whole edit format was 46% for the February model and only 39% for the June models.
But also note how much the solid green diff bars degrade between the February and June GPT-3.5 models. They drop from 30% down to about 19%.
I saw other signs of this degraded performance in earlier versions of the benchmark as well.
Recently people have claimed GPT4 is an ensemble model with 8 different models under the hood. My guess is that the "nerfing"(I've noticed it as well at random times) is when the model directs a question to the wrong underlying model
“Developers wishing to continue using their fine-tuned models beyond January 4, 2024 will need to fine-tune replacements atop the new base GPT-3 models (ada-002, babbage-002, curie-002, davinci-002), or newer models (gpt-3.5-turbo, gpt-4).“
They didn't mention gpt-4-32k. Does anybody know if it will be generally available in the same timeframe?
There's still no news about the multi-modal gpt-4. I guess the image input is just too expensive to run or it's actually not as great as they hyped it.
>I guess the image input is just too expensive to run or it's actually not as great as they hyped it.
We already know they have a SOTA model that can turn images into latent space vectors without being some insane resource hog - in fact, they give it away to competitors like Stability. [0]
My guess is a limited set of people are using the GPT-4 with CLIP hybrid, but those use-cases are mostly trying to decipher pictures of text (which it would be very bad at), so they're working on that (or other use-case problems).
Not a lot of talk of Whisper being available here.
From using voice in the ChatGPT iOS app, I surmise that Whisper is very good at working out what you've actually said.
But it's really annoying to have to say my whole bit before getting any feedback about what it's gonna think I said. Even if it's getting it right at an impressive rate.
Given this is how OpenAI themselves use it (say your whole thing before getting feedback), I don't know that the API is set up to be able to mitigate that at all, but it would be really nice to have something closer to the responsiveness of on-device dictation with the quality of Whisper.
I'm interested in how the transformer based speech recognition from iOS 17 will perform compared to Whisper. I guess it will work more "real-time" like the current dictation on iOS/macOS, but I'm unsure as I am not on the beta right now.
My guess is the reason that apple invested so heavily in this [0] is because they are going to train a big transformer in their datacenter and apply it as an RNN on your phone.
Superficially, I think this will work very well, but slightly worse than whisper (with the advantage ofc being that its better at real-time transcription).
I am a beta user and I am seriously impressed if I’m being honest. It works fully offline, is multilingual and is fast, even on the non-latest iPhone (I am using an iPhone 12 Pro Max). It is way better than apples previous version and better than locally installed whisper. They’ve done incredible work. Same with the new, transformer-based keyboard on iOS which is way better. And if you type in English, it sometimes shows word suggestions in the text field itself (similar how copilot works in an IDE).
One speculative thought about the purpose of Whisper is that this will help unlock additional high-quality training data that's only available in audio/video format.
Main reason for the lack of excitement is probably that it is fairly easy to self host Wisper, so people interested in it would have been doing that all along.
It's funny how OpenAI just shattered Google's PR stunts. Google wanted everyone to believe they are leading in AI by winning some children's games. Everyone thought that was the peak of AI. Enter OpenAI and Micorsoft. Microsoft and OpenAI have showed the humanity what true AI looks like. Like most people on HN I cannot wait to see the end of Google, the end of evil.
Yikes. They're actually killing off text-davinci-003. RIP to the most capable remaining model and RIP to all text completion style freedom. Now it's censored/aligned chat or instruct models with arbitrary input metaphor limits for everything. gpt3.5-turbo is terrible in comparison.
This will end my usage of openai for most things. I doubt my $5-$10 API payments per month will matter. This just lights more of a fire under me to get the 65B llama models working locally.
Please ELI5 if I am mis-interpretating what you said:
*"They have just locked down access to a model which they basically realized was way more valuable than even they thought - and they are in the process of locking in all controls around exploiting the model for great justice?"*
The Chat-GPT models are all pre-prompted and pre-aligned. If you work with davinci-003, it will never say things like, "I am an OpenAI bot and am unable to work with your unethical request"
When using davinci the onus is on you to construct prompts (memories) which is fun and powerful.
====
97% of API usage might be because of ChatGPT's general appeal to the world. But I think they will be losing a part of the hacker/builder ethos if they drop things like davinci-003, which might suck for them in the long run. Consumers over developers.
How do they want to commercialise it? Do they want moms to tinker on ChatGPT once a month to do their children's homework? Or do they want people to build businesses using their software
do they have the cash money dollar? and the willingness to spend it on what is essentially a toy they will quickly grow bored of? I don't think this is the best path to profitability
If you're using the API, you construct the "memories" as well, including the "system" prompt, even in the playground. (When you click the "(+) Add message", the new one defaults to USER, but you can click on it to change it to ASSISTANT, then fill it in with whatever you want.)
I used the "Complete" UI (from the Playground) for a bit before the "Chat" interface was available; I don't really think there's anything you couldn't do in the "Complete" UI that you couldn't also do in the "Chat" UI.
Note that the Azure endpoint is not being sunsetted until July 5th, 2024.
One supposes openai has a 6 month notice period vs a 12 month period for azure. This might generally effect one’s appetite in choosing which endpoint to use for any model.
> In the coming weeks, we will reach out to developers who have recently used these older models, and will provide more information once the new completion models are ready for early testing.
I wonder if there's some element of face-saving here to avoid a lawsuit that may come from someone that uses the model to perform negative actions. In general I've found that gpt3.5-turbo is better than text-davinci-003 in most cases, but I agree, it's quite sad that they're getting rid of the unaligned/censored model.
More likely hardware constraints. They can't get the hardware fast enough to do everything they want to do. So, they free up resources by ditching lower demand models.
Yeah, TextCompletion is much better than ChatCompletion with v3 models.
But with davinci at the same price point as GPT-4 I'm hoping the latter is enough of a step up in its variety of vocabulary and nudgeable sophistication of language to be a drop in replacement.
Though in general I think there's an under appreciation for just how much is being lost in the trend towards instruct models, and hope there will be smart actors in the market who use a pre-optimization step for instruct prompts that formats it for untuned models. I'd imagine that parameter size to parameter size that approach will look much more advanced to end users just by not lobotomizing the underlying model.
My guess is that they would be fine with continuing to serve all models, but that hardware constraints are forcing difficult decisions. SA has already said that hardware is holding them back from what they want to do. I was on a waiting list for the GPT4 API for like a few months, which I guess is because they couldn't keep up with demand.
I'm trying to create a bot that joins my friends Telegram group and melds into the conversation as if it was a real person. A real person might be the most cute and fun enthusiastic person there is but sometimes it has bad days, or it tells inappropriate jokes, right? People are complicated. Not this bot! No matter what prompt I'm using (with the chat API) it won't lose the happy happy joy joy chatGPT attitude, won't tell inappropriate jokes, won't give advice on certain topics and in general won't talk like a real person, not because of technological limitations.. You can feel it when it's just nerfed.
Trying the same prompts that gave nerfed "I am just an AI I can't speculate about the future" bs on completion API gave somewhat better results, but most of the time they were flagged as breaking the guidelines which is a TOS breach if done enough times.
This can be solved other than open models. The same thing happened with stable diffusion. Good thing it's open so you can still use the pre-nerfed 1.6 models.
I know it might be edgy or unpopular but I don't think one entity should decide how we can use this powerful tool. No matter its implications and consequences.
Or literally anything other than the psychotically smarmy tone of GPT-4 that's almost impossible to remove and constantly gives warnings, disclaimers and stops itself if veering even just 1 mm off the most boring status quo perspectives.
Lots of my favorite and frankly the best litterature in the world have elements that are obscene, grotesque, bizarre, gritty, erotic, frightening, alternative, provocative - but that's just too much for chat-gpt, instead it has this - in my eyes - way more horrifying smiling-borg-like nature with only two allowed emotions: "ultra happiness", and "ultra obedience to the status quo".
Anyone who doesn't has never actually toyed with LLM and received, "As an AI language model I can't" in response to an innocuous request to, say, write a limerick about a politician or list the reasons why "username2 is stupid".
But mostly it has to do with the fact that LLM do what they've seen. And if they've been fine-tuned to not respond to some classes of things they'll misapply that to lots of other things. That's why most people go for the "uncensored" fine tuning datasets for the llamas even for completely sfw use cases.
Note that code-davinci-002, despite the confusing name, is the actual GPT-3.5 base model, which only does completions and does not have any mode collapse. And it is still available via Azure, as far as I can tell. Text-davinci-003 is a fine-tuned version of it.
Practical report: the OpenAI API is a bad joke. If you think you can build a production app against it, think again. I've been trying to use it for the past 6 weeks or so. If you use tiny prompts, you'll generally be fine (that's why you always get people commenting that it works for them), but just try to get closer to the limits, especially with GPT-4.
The API will make you wait up to 10 minutes, and then time out. What's worse, it will time out between their edge servers (cloudflare) and their internal servers, and the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent.
Meanwhile, their status page will happily show all green, so don't believe that. It seems to be manually updated and does not reflect the truth.
Could it be that it works better in another region? Could it be just my region that is affected? Perhaps — but I won't know, because support is non-existent and hidden behind a moat. You need to jump through hoops and talk to bots, and then you eventually get a bot reply. That you can't respond to.
My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now.
There is no way to contact OpenAI, no way to report problems, the API sometimes kind-of works, but mostly doesn't, and if you comment in the developer forums, you'll mostly get replies from apologists that explain that OpenAI is "growing quickly". I'd say you either provide a production paid API or you don't. At the moment, this looks very much like amateur hour, and charging for requests that were never fulfilled seems like a fraud to me.
So, consider carefully whether you want to build against all that.
I’m only using them as a stop-gap / for prototyping with the intent to move to a locally hosted fine-tuned (and ideally 7B parameter) model further down the road.
> you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent.
Borderline!? They're regularly charging customers for products they know weren't delivered. That sounds like straight-up fraud to me, no borderline about it.
You mean it's not normal to tell people that it's their fault for driving their $80,000 electric car in heavy rain, because for many years you haven't bothered to properly seal your transmission's speed sensor?
I understand your general point and am sympathetic to it, if you're a 10/10 on some scale, I'm about a 3-4. I've never seen billings for failures, but the billing stuff is crazy: no stats if you do streamed chat, and the only tokenizer available is in Python and for GPT-3.0.
However, I'm virtually certain somethings wrong on your end, I've never seen a wait even close to that unless it was completely down. Also the thing about "small prompts"...it sounds to me like you're overflowing context, they're returning an error, and somethings retrying.
I can vouch on this. GPT4 API dies a lot if you use it for a big concurrent project. And of course it’s rate limited like crazy, with certain hours being so bad you can’t even run it for any business purpose.
Very sorry to hear about these issues, particularly the timeouts. Latency is top of mind for us and something we are continuing to push on. Does streaming work for your use case?
We definitely want to investigate these and the billing issues further. Would you consider emailing me your org ID and any request IDs (if you have them) at atty@openai.com?
Thank you for using the API, and really appreciate the honest feedback.
Quick note: your domain doesn't appear to have an A record. I was hoping to follow the link in your profile and see if you have anything interesting written about LLMs.
I know you guys are busy literally building the future but could you consider adding a search field in ChatGPT so that users can search their previous chats?
> We definitely want to investigate these and the billing issues further.
What’s a problem for OpenAI engineers to get web access logs and grep for 4xx/5xx errors?
It's kind of incredible how fast OpenAI (now also known as ClosedAI) is going through the enshittification process. Even Facebook took around a decade to reach this level.
OpenAI has an amazing core product, but in the span of six months:
* Went from an amazing and inspiring open company that even put "Open" in their name to a fully locked up commercial beast.
* Non-existent customers support and all kinds of borderline illegal billing practice. You guys are definitely aware that when there's a network error on the API or ChatGPT, the user still gets charged. And there's a lot of these errors. I get roughly one per hour or two.
* Frustratingly loose interpretation of EU data protection rules. For example, the setting to say "don't use my personal chat data" is connected to the setting to save conversations. So you can't disable it without losing all your chat history.
* Clearly nerfing the ChatGPT v4 products, at least according to hundreds or even thousands of commenters here and on reddit, while denying to have made any changes.
* Use of cheap human labor in developing countries through shady anonymous companies (look up the company Sama who pay Kenyan workers about $1.5 an hour).
* Not to mention the huge questions around the secret training dataset and whether large portions of it consist of illegally obtained private data (see the recent class court case in California)
> Use of cheap human labor in developing countries through shady anonymous companies (look up the company Sama who pay Kenyan workers about $1.5 an hour).
What is wrong about injecting millions into developing nations?
The rest I agree with, although I don't think it was ever really 'open' so its not getting shitty, it always was. Thankfully, "there is no moat" and other LLMs will be open, just a few months behind OpenAI
Since chatGPT-4 is now useless for advanced coding because of their blackbox sudden nerfing, can anyone guess how long before i can run something similar to the orig version privately?
Is the newer 64B models up there? 1 year, 2 years? Can't wait until i get back the crazy quality of the orig model.
We need something open source fast. Thanks open-ai for giving us a glimpse of the crazy possibilities, too crazy for the public i guess.
> * Use of cheap human labor in developing countries through shady anonymous companies (look up the company Sama who pay Kenyan workers about $1.5 an hour).
If you pay a developing country developed country wages what you'll get is 1. inflation and 2. the government mad at you because all their essential workers/doctors/government officials are quitting to work for you.
This is a terrible excuse that I see trotted out far to often to justify going to developing countries and barely even paying workers that country's minimum wage. You absolutely can pay considerably more than minimum wage without disrupting the local economy. They're paying people as low as $1.32 per hour for an absolutely horrible job. I'm not expecting them to pay western wages. But even bumping that up to $2.50 or $3 an hour would make an incredible difference to the local workers lives. The fact that they don't do that is exploitation, pure and simple.
Note that I feel I have quite deep understanding of this issue, and feel strongly about it, because I live and work in a developing country and I see this happening a lot. Westerners come over here and treat local workers like shit, pay them peanuts for 80 hour weeks while making loads of money themselves and then justify it because "it's the local norm". It's sickening, frankly. We westerners doing business in developing countries are in a position of privilege and should be leading by example, not jumping on the first excuse to dump a hundred years worth of the fight for workers rights.
I'm curious. When you buy a loaf of bread from the local market, are they cheaper than first world prices? If so, do you pay double the listed price and demand the shop pay double the price to hire workers so as to not exploit them? Are your expenses in said developing country lower than what you would have paid if you were in a richer country? Are you donating the difference to the local community?
It's weird how people always trot out phrases like "I'm just curious" or "I'm just asking questions here" when they try to justify exploitation. Is it so that you have plausible deniability when you inevitably get called on it? Because that doesn't work.
I see that you have pretty extreme takes on what constitutes "exploitation". It's one thing to pretend that you're not part of it if you live on another part of the planet and pretend globalization doesn't exist, but I was wondering how you'd avoid participating in it if you lived in the same country and economic bubble as the ones you claim are exploited.
If you had a morally consistent way to live that life, you'd have my respect. But no, you had to deflect the topic to a phrase I wrote and make presumptions about what I really meant.
FYI, I'm morally at ease with myself, I don't need to justify anything to anyone.
Hi, I've been to Kenya and Tanzania, and while basic staples are cheaper than developed countries they're not that much cheaper these days. If you watch travelog videos where they ask locals how they're doing, many developing countries are struggling with massive inflation that's been partly caused by volatile energy prices (many people can no longer afford gas) and partly by food shortages from the Ukraine War.
Not to nitpick, but if you're able to name the company employing Kenyans, Sama, who's homepage is at https://www.sama.com/, with a team page at https://www.sama.com/our-team/ , I'm not sure you can complain that they're being shady and anonymous.
It's pretty shady. They have been fully exposed at this stage but from what I understand they were trying to keep a very low profile, going to efforts to make sure the Kenyan workers didn't know they were working for a company called Sama but instead using sub companies to sign the worker contracts.
They also no longer support data exports for many users (including myself) - at one point it worked but now it says you'll receive an email to download your data, which never arrives.
While you're here, you should know that the logic for enabling GPT-4 API access is excluding Microsoft for Startups (https://openai.com/microsoft-for-startups) orgs which have valid billing against Microsoft-provided credits. Presumably, this is an oversight as it wouldn't make sense to exclude pre-existing Microsoft partners. Would you mind escalating this?
This is precisely what is wrong with OpenAI. This. Right here.
"Complaining on HN will get you access. You have know people or "complain in the right forums."
THERE SHOULD BE NO LOGIC.
No qualifying rules. No access checks. No gates. No hoops.
Sam Altman has gone on a worldwide interview tour claiming he wants to "democratise access to AI", meanwhile OpenAI is the least open company I have ever dealt with, or even heard of.
Oracle is more "democratic" and open, for crying out loud.
Streaming has no value for me, I need the entire response before I can do anything. I looked at streaming, but it seemed like a significant effort to implement, and with no obvious benefit — if I get 75% of my response through streaming and then something breaks, it doesn't get me anywhere.
Thanks for offering help, I will contact you directly.
After one of the ubuntu snap updates my firefox stopped working with OpenAI API playground it worked still with every other site. I retried and restarted so many times and it didn't work. Eventually I switched browser to chromium and it worked. I still don't know the problem and it was unnerving, I would have a lot of anxiety to build something important with it.
I tried again just now and I got "Oops! We ran into an issue while authenticating you." but it works on chromium.
Lmao. You had a browser issue when running Firefox on Linux (.000001% of users) and now you are making connections between that and their API stability?
I’m pretty sure they tuned the Cloudflare WAF rules on GPT 3 and forgot to increase the request size limits when they added the bigger models with longer contest windows.
Probably compute-bound for inference which they've probably built in an arch-specific way, right? This sort of thing happens. You can't use AVX-512 in Alibaba Cloud cn-hongkong, for instance, because there's no processor available there that can reliably do that (no Genoa CPUs there). I imagine OpenAI has a similar constraint here.
I've heard LLMs described as "setting money on fire" from people that work in the actually-running-these-things-in-prod industry. Ballpark numbers of $10-20/query in hardware costs. Right now Microsoft (through its OpenAI investment) and Google are subsidizing these costs, and I've heard it's costing Microsoft literally billions a year. But both companies are clearly betting on hardware or software breakthroughs to bring the cost down. If it doesn't come down there's a good chance that it'll remain more economical to pay someone in the Philippines or India to write all the stuff you would have ChatGPT write.
yeah this isnt close. Sam Altman is on record saying its single digit cents per query and then took a massively dilutive $10b investment from microsoft. Even if gpt4 is 8 models in a trenchcoat they wouldnt raise it on themselves by 4 orders of magnitude like that
Single digit cents per query (let's say 2) is A LOT. Let's say the service runs at 10krps (made up, we can discuss about this) it means the service costs 200$ a second i.e 20M$ a day (oversimplifying a day with 100k seconds, but this might be ok to get us in the ballpark), which means that running the model for a year (400 days, sorry simplifying) is around 8B$, so too run 10krps we are in the order of billions per year. We can discuss some of the assumptions but I think that of we are in the ballpark of cents per query the infrastructure costs are significant.
Note that /r/ChatGPT is mostly nontechnical people using the web UI, not developers using the API.
It's very possible the web UI is using a nerfed version of the model evident by its different versioning, but not the API which has more distinct versioning.
> the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent.
It's fraudulent, full stop. Maybe they're able to weasel out of it with credit card companies because you're buying "credits."
I suspect it was done this way out of pure incompetence; the OpenAI team handling the customer-facing infrastructure have a pretty poor history. Far as I know you still can't do something simple like change your email address.
In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/
You should apply and use OpenAI on azure. We’ve got close to 1m tokens per minute capacity across 3 instances and the latency is totally fine, like 800ms average (with big prompts). They’ve just got the new 0613 models as well (they seem to be about 2 weeks behind OpenAI). We’ve been in production for about 3 months, have some massive clients with a lot traffic and our gpt bill is way under £100 per month. This is all 3.5 turbo though, not 4 (but that’s available on application, but we don’t need it).
I built a production app on top of OpenAI and yeah there are frequent errors and timeouts but you literally just have to add some code to account for these and it works fine...
For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up
It is possible to have a stable production app on top of it
Just fix your code and stop expecting OpenAI to hold your hand
> Just fix your code and stop expecting OpenAI to hold your hand
:-)
My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not.
At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered.
Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
The difference between 4 and 3.5 is really big for creative use cases. I am running an app with significant traffic and the retention of users on GPT-4 is much higher.
Unfortunately it's still too expensive and the completion speed is not as high as GPT-3.5 but I hope both problems will improve over time.
561 comments
[ 2.8 ms ] story [ 327 ms ] threadThe mass adoption of the ChatGPT APIs compared to the old Completion APIs proves my initial blog post on the ChatGPT API correct: developers will immediately switch for a massive price reduction if quality is the same (or better!): https://news.ycombinator.com/item?id=35110998
However if you try the gpt-4 API, it's possible it will be much better.
So still waiting to be on the same 32 pages...
Medical writing is the opposite, with unstated premises, semi-random associations, and rarely a meaningful sentence.
That makes sense. The labor impact research suggests that law will be a domain hit almost as hard as education by language models. Almost nothing happens in court that hasn't occured hundreds of thousands of times before. A model with GPT-4 power specifically trained for legal matters and fine tuned by jurisdiction could replace everyone in a courtroom. Well there's still the bailiff, I think that's about 18 months behind.
I built notionsmith.ai on 3.5: for some time I experimented with GPT 4 but the result was significantly worse to use because of how slow it became, going from ~15 seconds per generated output to a minute plus.
And you could work around that with things like streaming output for some use cases, but that doesn't work for chain of thought. GPT 4 can do some tasks without chain of thought that 3.5 required it for, but there are still many times where it improves the result from 4 dramatically.
For example, I leverage chain of thought in replies to the user when they're in a chat and that results in a much better user experience: It's very difficult to run into the default 'As a large language model' disclaimer regardless of how deeply you probe a generated experience when using it. GPT 4 requires the same chain of thought process to avoid that, but ends up needing several seconds per response, as opposed to 3.5 which is near-instant.
-
I suspect a lot of people are building things on 4 but would get better quality of output if they used more aspects of chain of thought and either settled for a slower output or moved to 3.5 (or a mix of 3.5 and 4)
For instance, with MATH dataset, my own n=500 evaluation showed no difference between GPT-3.5 (w/ and w/o CoT) and GPT-4. I was pretty surprised by that.
My project takes transcripts from YouTube, which don't have punctuation, splits them up into chunks, and passes each chunk to GPT-4 telling it to add punctuation with paragraphs. Part of the instructions includes telling the model that, if the final sentence of the chunk appears incomplete, to just try to complete it. Anyway, GPT-3.5-turbo works okay for several chunks but almost invariably hits a case where it either writes a bunch of nonsense or spells out the individual letters of words. I'm sure that there's a programmatic way I can work around this issue, but GPT-4 performs the same job flawlessly.
And what open LLM you used it with / how successful you've found it?
poe law
I mostly use it for generating tests, making documentation, refactoring, code snippets, etc. I use it daily for work along with copilot/x.
In my experience GPT3.5turbo is... rather dumb in comparison. It makes a comment explaining what a method is going to do and what arguments it will have - then misses arguments altogether. It feels like it has poor memory (and we're talking relatively short code snippets, nothing remotely near it's context length).
And I don't mean small mistakes - I mean it will say it will do something with several steps, then just miss entire steps.
GPT3.5turbo is reliably unreliable for me, requiring large changes and constant "rerolls".
GPT3.5turbo also has difficulty following the "style/template" from both the prompt and it's own response. It'll be consistent then just - change. An example being how it uses bullet points in documentation.
Codex is generally better - but noticeably worse then GPT4 - it's decent as a "smart autocomplete" though. Not crazy useful for documentation.
Meanwhile GPT4 generally nails the results, occasionally needing a few tweaks, generally only with long/complex code/prompts.
tl;dr - In my experience for code GPT3.5turbo isn't even worth the time it takes to get a good result/fix the result. Codex can do some decent things. I just use GPT4 for anything more then autocomplete - it's so much more consistent.
Where 3.5 excels is with programmatic access. You can ask it for 2x as much text between setup so the end result is well formed and still get a reply that's cheaper and faster than 4 (for example, ask 3.5 for a response, then ask it to format that response)
nooooo they are deprecating the remnants of the base models
The chat models constantly argue with you on certain tasks and are highly opinionated. A completion API was a lot more flexible and "vanilla" about a wide variety of tasks, you could start a thought, or a task, and truly have it complete it.
The chat API doesn't complete, it responds (I mean of course internally it completes, but completes a response, rather than a continuation).
I find this a big step back, I hope the competition steps in to fill the gaps OpenAI keeps opening.
Comparable, perhaps.
OpenAI will never fulfill the entire market, and their moat is in danger with every other company that has LLM cash flow.
They want to become the AWS of AI, but it's becoming clear they'll lose generative multimedia. They may see the LLM space become a race to the bottom as well.
I made up an example here to illustrate, but it's just very annoying because sometimes it puts at the beginning, slowing down my interaction, and it now refuses to obey my prompts to leave caveats out.
https://chat.openai.com/share/1f39af02-331d-4901-970f-2f4b0e...
looking at your sample and using character count as a rough proxy for tokens, (465/(1581-465))*100 means they added ~42% token count cost to your response explicitly adding caveats which you dont want. fun!
GPT-4 is on a completely different level of consistency and actually listening to your system prompt than chagpt-3.5. It trails off much more rarely.
If only it wasn't so slow/expensive... (it really starts to hurt with large token counts).
I am especially interested in gpt-3.5-turbo-instruct, as I think that the hype surrounding ChatGPT and "conversational LLMs" has sucked a lot of air out of what is possible with general instruct models. Being able to fine tune it will be phenomenal as well.
I do not really understand the efforts that went on behind the scenes to train GPT models on factual data. Did humans have to hand approve/decline responses to increase its score?
"America is 49 states" - decline
"America is 50 states" - approve
Is this how it worked at a simple overview? Do we know if they are working on adding the rest of 2021, then 2022, and eventually 2023? I know it can crawl the web with the Bing addon but, it's not the same.
I asked it about Maya Kowalski the other day. Sure it can condense a blog post or two, but it's not the same as having the intricacies as if it actually was trained/knew about the topic.
Has anyone fared better? I might be doing something wrong but I can't see what that could possibly be.
We use it to generate automatic insights from survey data at a weekly cadence for Zigpoll (https://www.zigpoll.com). This makes getting an instant response unnecessary but still provides a lot of value to our customers.
> We recognize this is a significant change for developers using those older models. Winding down these models is not a decision we are making lightly. We will cover the financial cost of users re-embedding content with these new models. We will be in touch with impacted users over the coming days.
You don't need very large models to generate usable embeddings.
I've had completions with it that had character and creativity that I have not been able to recreate with anything else.
Brilliant and hilarious things that are a permanent part of my family's cherished canon.
https://raw.githubusercontent.com/thomasdavis/omega/master/s...
Had it hooked up to speech so you could just talk at it and it would talk back at you.
Gave incredible answers that ChatGPT just doesn't do at all.
I've been working on LLMs for creative tasks and believe a mix of chain of thought and injecting stochasticity (like instructing the LLM to use certain random letters pulled from an RNG in a certain way at certain points) can go a long way in terms of getting closer to human-like creativity
It relies on WebSockets, Js, and a reasonably stable connection to run since it's built on Blazor
gpt-3.5-turbo is the model behind ChatGPT. It's chat-fine-tuned which makes it very hard to use for use-cases where you really just want it to obey/complete without any "chatty" verbiage.
The "davinci-003" model was the last instruction tuned model, but is 10x more expensive than gpt-3.5-turbo, so it makes economical sense to hack gpt-3.5-turbo to your use case even if it is hugely wasteful from a tokens point of view.
The difference between them is that the chat models are much more... chatty - they're trained to act like they're in a conversation with you. The chat models generally say things "Sure, I can do that for you!", and "No problem! Here is". The conversation style is generally more inconsistent in it's style. It can be difficult to make it only return the result you want, and occasionally it'll keep talking anyway. It'll also talk in first person more, and a few things like that.
So if you're using it as an API for things like summarization, extracting the subject of a sentence, code editing, etc, then the chat model can be super annoying to work with.
Have you tried getting your formatted JSON out via the new Functions API? I does cure a lot of the deficiencies in 3.5-turbo.
https://openai.com/pricing
https://platform.openai.com/docs/deprecations/
Think of the properties you want in the JSON object, then send those to ChatGPT as required parameters for a function (even if that function doesn't exist).
I've found it's not perfect but still pretty reliable – good enough for me combined with error handling.If you're interested, I wrote a blog post with more detail: https://puppycoding.com/2023/07/07/json-object-from-chatgpt-...
https://github.com/microsoft/guidance
There was no question something happened in January with ChatGPT, weirdly would refuse to answer questions that were harmless but difficult(Give me a daily schedule of a stoic hedonist)
Every once in a while, I see redditors complain of it being nerfed.
Sometimes I go back to gpt3.5 and am mind boggled how much worse it is.
Makes me wonder if they keep increasing the version number while dumbing down the previous model.
With an API, being unreliable would be a deal-breaker. Looking forward to people fine-tuning LLMs with GPT4 API. I'd love it for medical purposes, I'm so worried of a future where the US medical cartels ban ChatGPT for medical purposes. At least with local models, we don't have to worry about regression.
The current versions is close to the original 3.5 version, while 3.5 has become horribly bad, such a scam to not disclose what's going on, especially for a paid service.
The fact that rumors about GPT-5 were quickly suppressed and the models were dumbed down even more cannot be entirely explained by excessive demand. I think it's more likely that GPT-3.5 and GPT-4 demonstrated unexpected capabilities in the hands of the public leading to a pull back. Moreover, Sam Altman's behaviors changed dramatically between the initial release and a few weeks afterward -- the extreme optimism of a CEO followed by a more subdued, even cowed, demeanor despite strong enthusiasm from end-users.
OpenAI cannot do anything without Microsoft's data center resources, and Microsoft is a critical defense contractor.
Anyway, personally, I'm with the crowd that thinks we're about to see a Cambrian explosion of domain-specific expert AIs. I suspect that OpenAI/Microsoft/Gov is still trying to figure out how much to nerf the capability of GPT-3.5 to tutor smaller models (see "Textbooks are all you need") and that's why the API is trash.
the one from a MS researchers that has been recommended to all of us probably. Good video btw.
My guess is that they began to restrict ChatGPT because they can't sell that. They probably want to sell you CodeGPT or other products in the future so why would they give that away for free? ChatGPT is just a teaser.
This is my experience also. I have not formally benchmarked the different releases, but specifically for Python coding ChatGPT 4 got considerably worse with the latest updates.
Is there any actual evidences other than some user subjective experiences?
https://news.ycombinator.com/item?id=36179783
One would pay because what they get out of chatGPT provides value, of course. Keep in mind that the users of these 2 products can be (and in fact are) different — chatGPT is a lot friendlier (from a UX perspective) than using the API playground (or using the API itself).
They are killing text-davinci-003 btw.
[2]: https://www.youtube.com/watch?v=qbIk7-JPB2c&t=1392s
[1]: https://arxiv.org/abs/2303.12712
Noobie follow up question: Should we put any trust into “Sparks of intelligence” I thought it was regarded as a Microsoft marketing piece, not a serious paper.
Also notable that the team behind that paper wasn't involved in designing/building the model, but they did get access to prerelease versions.
This is the double edge sword of being so ridiculously closed.
[0] https://en.m.wikipedia.org/wiki/The_Market_for_Lemons
You can apparently have it be nice or smart, but not both.
Even on this forum I've seen AI enthusiasts claiming AI will be the best psychologist, best school teacher, etc.
https://news.ycombinator.com/item?id=35402777
https://www.businessinsider.com/widow-accuses-ai-chatbot-rea...
buffing is the opposite, when an item gets better.
?
https://chat.openai.com/share/04c1dbc0-4890-447f-b5a5-7b1bc5...
https://aider.chat/docs/benchmarks.html
But unfortunately, you have to read a ways into the doc and understand a lot of details about the benchmark. Here's a direct link and excerpt of the relevant portion:
https://aider.chat/docs/benchmarks.html#the-0613-models-seem...
The benchmark results have me fairly convinced that the new gpt-3.5-turbo-0613 and gpt-3.5-16k-0613 models are a bit worse at code editing than the older gpt-3.5-turbo-0301 model.
This is visible in the “first attempt” portion of each result, before GPT gets a second chance to edit the code. Look at the horizontal white line in the middle of the first three blue bars. Performance with the whole edit format was 46% for the February model and only 39% for the June models.
But also note how much the solid green diff bars degrade between the February and June GPT-3.5 models. They drop from 30% down to about 19%.
I saw other signs of this degraded performance in earlier versions of the benchmark as well.
So need to pay to fine tune again?
There's still no news about the multi-modal gpt-4. I guess the image input is just too expensive to run or it's actually not as great as they hyped it.
https://help.openai.com/en/articles/7102672-how-can-i-access...
The decision of burying these extra information in a support article, not cool!
We already know they have a SOTA model that can turn images into latent space vectors without being some insane resource hog - in fact, they give it away to competitors like Stability. [0]
My guess is a limited set of people are using the GPT-4 with CLIP hybrid, but those use-cases are mostly trying to decipher pictures of text (which it would be very bad at), so they're working on that (or other use-case problems).
[0]https://github.com/openai/CLIP
Things that GPT-4 would easily, and correctly, reason through in April/May it just doesn't do any longer.
From using voice in the ChatGPT iOS app, I surmise that Whisper is very good at working out what you've actually said.
But it's really annoying to have to say my whole bit before getting any feedback about what it's gonna think I said. Even if it's getting it right at an impressive rate.
Given this is how OpenAI themselves use it (say your whole thing before getting feedback), I don't know that the API is set up to be able to mitigate that at all, but it would be really nice to have something closer to the responsiveness of on-device dictation with the quality of Whisper.
Using built-in text input showed quite good results since ChatGPT is still understanding the ask quite well
Pretty crazy stuff — perfectly understandable translations.
Superficially, I think this will work very well, but slightly worse than whisper (with the advantage ofc being that its better at real-time transcription).
[0]https://machinelearning.apple.com/research/attention-free-tr...
What is the difference? Replacing evil with another evil.
This is just behemoths exchanging hands.
This will end my usage of openai for most things. I doubt my $5-$10 API payments per month will matter. This just lights more of a fire under me to get the 65B llama models working locally.
*"They have just locked down access to a model which they basically realized was way more valuable than even they thought - and they are in the process of locking in all controls around exploiting the model for great justice?"*
What are funs things we can with it until it sunsets on January 4, 2024?
When using davinci the onus is on you to construct prompts (memories) which is fun and powerful.
====
97% of API usage might be because of ChatGPT's general appeal to the world. But I think they will be losing a part of the hacker/builder ethos if they drop things like davinci-003, which might suck for them in the long run. Consumers over developers.
I used the "Complete" UI (from the Playground) for a bit before the "Chat" interface was available; I don't really think there's anything you couldn't do in the "Complete" UI that you couldn't also do in the "Chat" UI.
One supposes openai has a 6 month notice period vs a 12 month period for azure. This might generally effect one’s appetite in choosing which endpoint to use for any model.
I guess they'll give you early access to it.
But with davinci at the same price point as GPT-4 I'm hoping the latter is enough of a step up in its variety of vocabulary and nudgeable sophistication of language to be a drop in replacement.
Though in general I think there's an under appreciation for just how much is being lost in the trend towards instruct models, and hope there will be smart actors in the market who use a pre-optimization step for instruct prompts that formats it for untuned models. I'd imagine that parameter size to parameter size that approach will look much more advanced to end users just by not lobotomizing the underlying model.
Trying the same prompts that gave nerfed "I am just an AI I can't speculate about the future" bs on completion API gave somewhat better results, but most of the time they were flagged as breaking the guidelines which is a TOS breach if done enough times.
This can be solved other than open models. The same thing happened with stable diffusion. Good thing it's open so you can still use the pre-nerfed 1.6 models.
I know it might be edgy or unpopular but I don't think one entity should decide how we can use this powerful tool. No matter its implications and consequences.
FOS for the win.
Lots of my favorite and frankly the best litterature in the world have elements that are obscene, grotesque, bizarre, gritty, erotic, frightening, alternative, provocative - but that's just too much for chat-gpt, instead it has this - in my eyes - way more horrifying smiling-borg-like nature with only two allowed emotions: "ultra happiness", and "ultra obedience to the status quo".
But mostly it has to do with the fact that LLM do what they've seen. And if they've been fine-tuned to not respond to some classes of things they'll misapply that to lots of other things. That's why most people go for the "uncensored" fine tuning datasets for the llamas even for completely sfw use cases.
More info:
https://platform.openai.com/docs/model-index-for-researchers
The API will make you wait up to 10 minutes, and then time out. What's worse, it will time out between their edge servers (cloudflare) and their internal servers, and the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent.
Meanwhile, their status page will happily show all green, so don't believe that. It seems to be manually updated and does not reflect the truth.
Could it be that it works better in another region? Could it be just my region that is affected? Perhaps — but I won't know, because support is non-existent and hidden behind a moat. You need to jump through hoops and talk to bots, and then you eventually get a bot reply. That you can't respond to.
My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now.
There is no way to contact OpenAI, no way to report problems, the API sometimes kind-of works, but mostly doesn't, and if you comment in the developer forums, you'll mostly get replies from apologists that explain that OpenAI is "growing quickly". I'd say you either provide a production paid API or you don't. At the moment, this looks very much like amateur hour, and charging for requests that were never fulfilled seems like a fraud to me.
So, consider carefully whether you want to build against all that.
https://learn.microsoft.com/en-us/azure/cognitive-services/o...
Borderline!? They're regularly charging customers for products they know weren't delivered. That sounds like straight-up fraud to me, no borderline about it.
I meant it's not normal to start selling a feature in 2016 and delivering it in beta seven years later.
If you want better latency and sane billing you need to go through Azure OpenAI Services.
OpenAI also offers decreased latency under the Enterprise Agreement.
However, I'm virtually certain somethings wrong on your end, I've never seen a wait even close to that unless it was completely down. Also the thing about "small prompts"...it sounds to me like you're overflowing context, they're returning an error, and somethings retrying.
https://news.ycombinator.com/newsguidelines.html
Very sorry to hear about these issues, particularly the timeouts. Latency is top of mind for us and something we are continuing to push on. Does streaming work for your use case?
https://github.com/openai/openai-cookbook/blob/main/examples...
We definitely want to investigate these and the billing issues further. Would you consider emailing me your org ID and any request IDs (if you have them) at atty@openai.com?
Thank you for using the API, and really appreciate the honest feedback.
OpenAI has an amazing core product, but in the span of six months:
* Went from an amazing and inspiring open company that even put "Open" in their name to a fully locked up commercial beast.
* Non-existent customers support and all kinds of borderline illegal billing practice. You guys are definitely aware that when there's a network error on the API or ChatGPT, the user still gets charged. And there's a lot of these errors. I get roughly one per hour or two.
* Frustratingly loose interpretation of EU data protection rules. For example, the setting to say "don't use my personal chat data" is connected to the setting to save conversations. So you can't disable it without losing all your chat history.
* Clearly nerfing the ChatGPT v4 products, at least according to hundreds or even thousands of commenters here and on reddit, while denying to have made any changes.
* Use of cheap human labor in developing countries through shady anonymous companies (look up the company Sama who pay Kenyan workers about $1.5 an hour).
* Not to mention the huge questions around the secret training dataset and whether large portions of it consist of illegally obtained private data (see the recent class court case in California)
What is wrong about injecting millions into developing nations?
The rest I agree with, although I don't think it was ever really 'open' so its not getting shitty, it always was. Thankfully, "there is no moat" and other LLMs will be open, just a few months behind OpenAI
Please don't try to reframe this to make exploitation a positive thing. See my other comment here.
https://news.ycombinator.com/item?id=36625438
Is the newer 64B models up there? 1 year, 2 years? Can't wait until i get back the crazy quality of the orig model.
We need something open source fast. Thanks open-ai for giving us a glimpse of the crazy possibilities, too crazy for the public i guess.
If you pay a developing country developed country wages what you'll get is 1. inflation and 2. the government mad at you because all their essential workers/doctors/government officials are quitting to work for you.
Note that I feel I have quite deep understanding of this issue, and feel strongly about it, because I live and work in a developing country and I see this happening a lot. Westerners come over here and treat local workers like shit, pay them peanuts for 80 hour weeks while making loads of money themselves and then justify it because "it's the local norm". It's sickening, frankly. We westerners doing business in developing countries are in a position of privilege and should be leading by example, not jumping on the first excuse to dump a hundred years worth of the fight for workers rights.
Just curious.
If you had a morally consistent way to live that life, you'd have my respect. But no, you had to deflect the topic to a phrase I wrote and make presumptions about what I really meant.
FYI, I'm morally at ease with myself, I don't need to justify anything to anyone.
https://time.com/6247678/openai-chatgpt-kenya-workers/
"Complaining on HN will get you access. You have know people or "complain in the right forums."
THERE SHOULD BE NO LOGIC.
No qualifying rules. No access checks. No gates. No hoops.
Sam Altman has gone on a worldwide interview tour claiming he wants to "democratise access to AI", meanwhile OpenAI is the least open company I have ever dealt with, or even heard of.
Oracle is more "democratic" and open, for crying out loud.
Thanks for offering help, I will contact you directly.
I tried again just now and I got "Oops! We ran into an issue while authenticating you." but it works on chromium.
I’m pretty sure they tuned the Cloudflare WAF rules on GPT 3 and forgot to increase the request size limits when they added the bigger models with longer contest windows.
as far as I know OpenAI only has one region, that is out in Texas.
even more hilariously, as far as I can tell, Azure OpenAI -also- only has one region.. cant imagine why
https://learn.microsoft.com/en-us/azure/cognitive-services/o...
It's definitely limited, but there's currently more than one region available.
(I happen to be working at the moment on a location-related fix to our most popular Azure OpenAI sample, https://github.com/Azure-Samples/azure-search-openai-demo )
I too had an issue and put in a request. Took about 2.5 months to get a response, so 5 weeks you are almost half way there.
https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jq...
I've heard LLMs described as "setting money on fire" from people that work in the actually-running-these-things-in-prod industry. Ballpark numbers of $10-20/query in hardware costs. Right now Microsoft (through its OpenAI investment) and Google are subsidizing these costs, and I've heard it's costing Microsoft literally billions a year. But both companies are clearly betting on hardware or software breakthroughs to bring the cost down. If it doesn't come down there's a good chance that it'll remain more economical to pay someone in the Philippines or India to write all the stuff you would have ChatGPT write.
If you assume an A100 lives at the cutting edge for 2 years, that's about a million minutes, or $0.01 per minute of amortized HW cost.
In the crazy scenarios, I've heard 10 A100s per query, so assuming that takes a minute, maybe $0.1 per query.
Add an order of magnitude on top of that for labor/networking/CPU/memory/power/utilization/general datacenter stuff, you get to maybe $1/query.
So probably not $10, but maybe if you amortize training, low to mid single digits dollars per query?
It's very possible the web UI is using a nerfed version of the model evident by its different versioning, but not the API which has more distinct versioning.
It's fraudulent, full stop. Maybe they're able to weasel out of it with credit card companies because you're buying "credits."
I suspect it was done this way out of pure incompetence; the OpenAI team handling the customer-facing infrastructure have a pretty poor history. Far as I know you still can't do something simple like change your email address.
In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/
For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up
It is possible to have a stable production app on top of it
Just fix your code and stop expecting OpenAI to hold your hand
:-)
My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not.
At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered.
Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
We spend over $20k/mo with them and don't have any issues like this.
12 minute timeouts make no sense because first of all why are you even waiting 12 minutes?
Get off HN and go fix your shit
Unfortunately it's still too expensive and the completion speed is not as high as GPT-3.5 but I hope both problems will improve over time.