> As a result, Indeed was able to improve cost and latency by reducing the number of tokens in prompt by 80%.
There's some exact words shenanigans here. Indeed may have reduced the number of tokens in the prompt by 80%, but they didn't reduce the cost by 80%: the prompt cost of inferring from a fine-tuned GPT-3.5-turbo ($3.00 / 1M tokens) is 6x the prompt cost of inferring from the base GPT-3.5-turbo ($0.50 / 1M tokens). If prompt tokens are cut to 20% of normal, then that implies the overall cost of the prompt tokens is 120% relative to their normal prompt: a cost increase! That's not even getting into the 4x cost of the completion tokens for a finetuned model.
Of course, Indeed likely has an enterprise contract reducing costs further.
The "bubble" you're speaking of now is completely divorced from reality in both directions, as the doomer narrative spread across the blogosphere/news world with the recent inflection talent acquisition and the issues with stability.ai/emad.
The truth is that foundation model companies, while in the limelight now, are probably the smallest slice of the pie, with cloud providers and end user applications going to take the lion share of the profit, and GPU manufacturers in line next. Foundation model companies got hyped because every VC wanted to invest in the company that might monopolize AGI, but in reality we're going to have a landscape dominated by fine tuned open source models in a few years, with closed source models being used for certain niches but generally not worth the cost.
Number of tokens is still a useful metric, as their endpoints have Tokens Per Minute quotas. Decreasing number of tokens used means increasing throughput, up until the Request Per Minute quota.
I was somewhat involved in this project. Can't get into details but there were other factors/efforts not mentioned which allowed us to scale this while reducing cost per recommendation. As someone mentioned, I do believe we benefited from a price drop over time.
Regarding the monthly scale mentioned in article–we are way beyond that now.
A lot of really smart people worked on this and it was fun to watch unfold.
Can't wait till I have to pay a lawyer $500/hr to turn around and ask a bot what to do.
Given the disproportionate amount of lawyers in any given legislature, never mind the make-up of judges, I can only imagine that a legal stranglehold insulating lawyers from the threat of AI will be one of, if not the, first bit of AI protectionist legislation.
Computers can never be responsible or accountable for their actions. AI is like the genie from the lamp, it will grant your 3 wishes and they will turn out weird. And you can't do anything to it. What can you do, it has no body. Only a human has skin in the game.
Trust me, existing LLMs are nowhere near being able to provide that kind of legal analysis. With the diminishing rate of improvement in the frontier models that we’ve seen, I’m also doubtful this particular technology is on a path to get there.
OpenAI are pretty much the most dominant LLM company at the moment.
I just wish they let you make new accounts after deleting an old account, apparently it's something to do with the database when I ask them about it, which suggests that they store data about users even after someone deletes their account.
These people can’t just lay low for even a few days. It’s been one reminder after another that for some almost certainly bad reason, OpenAI wants everyone to remember that some of the most scandalous revelations with a mountain of even more damning and extremely credible allegations hot on their heels is a new record for SV insanity, at least since the early 90s that I’ve been watching.
The CEO was (up to a credible consensus of primary sources) fired for self-dealing in borderline criminal fraud-level ways. I’m not aware of anyone disputing this with any vigor. Then that CEO was fired again for more vague but thematically consistent allegations of flagrantly profiteering off an ostensible charity actively lobbying for de-facto regulator status. Then that CEO made a token concession by disposing of yet another in this cavalcade of personal financial gain presented as urgent imperative.
And the most moderating influence is a guy who has been run out of everything from politics to finance to prestigious academic posts over scandal after scandal in print for 35 years running, and has yet to issue an apology I’m aware of.
But the really high thread-count noose is that Opus is mopping the floor with GPT-4-1106 and 0125, on money that would fall out of Nadella’s couch cushions. And it’s dramatically operator-aligned.
You can’t be a society-scale criminal, keep it up for over a decade, and get lapped by vastly less-resourced competitors in the Valley. You shouldn’t be able to do any of that and still wield any power.
This new breed of tech titans have forgotten something their predecessors knew: it took Wall Street a century to utterly capture its regulators.
Tech people who go big enough? Holmes is serving time, SBF is serving time, they’re not the first.
This isn’t about corrupting the competitive landscape anymore geniuses: this is about staying out of prison.
Altman/Loopt, Siebel/Socialcam, Emmett/Twitch failed and got rich. The people who trusted them lost a lot. People still say they’re smart.
Larry said we should put as much poison in Africa as we can. He wrote it down. He tried to get someone fired, by ratting out his friends. He made bankers angry. They never forgave him. So he moved. Also, Brooksley was going to save America. Larry said she was a woman and be quiet. Also stealing is ok.
Bankers don’t go to prison. They can do anything. But it’s not because they’re rich. It’s because they know important people. Also rich.
Creepy tech people aren’t like bankers. They don’t get in trouble much. But if they’re bad enough, they go to jail.
Someone did the worst thing in a long time. They’ve done it a lot.
He should say sorry and get a different job, because jail sucks.
A little bit hard to see how this is relevant to a thread about Introducing improvements to the fine-tuning API and expanding our custom models program.
The person you're replying to can correct me if I'm wrong, but I think what they were originally implying with "laying low" is that OpenAI has a history of trying to bury press by releasing fancy new features and remarkable improvements with impeccable timing.
This sounds interesting enough. Could I have it as a more detail text based novel (150-250 pages) or if it's possible a full fledged movie, maybe in similar style as "Big Short"?
Or do you think it has to be a TV show, maybe like "Succession" - do you think you'd be able to write something like "Succession"?
I’m really not sure what you think you’re adding to the conversation with these flippant remarks, but it’s unfortunate that you seem to think it’s an appropriate response to the well-thought-out criticisms that benreesman put together.
You might think it's flippant remarks or a sarcastic tone, which I can understand, but I was honestly imagining in my head what a film about all of this would look like. I would honestly, even desperately like to know what is going on in a head of Sam Altman, or if this isn't possible, then any interpretation of what someone else might think that is going on in his head. So while I understand how I am perceived like that, I'm at the same time really curious about the topic, and actually really curious about the output of the OP. When I mentioned the 150 - 250 page novel, I was actually really desiring to read that, because OP had such unique perspective, I just needed to know more.
I was specifically calculating the 150 - 250 pages to possibly contain enough detail to have many of the explanations that I would have energy to read.
Unfortunately, I'm not going to get this book, so I will have to deal with my everyday life instead.
I was kind of hoping that mentioning "Succession" due to its nature would give it away, that I'm respecting the intelligence of the criticism, because I feel like "Succession" is a really intelligent and insightful show.
Obviously the way I commented it, it didn't come off like that and I was trying to be unnecessarily funny when I shouldn't have been, but I don't 100% regret it, I did it, there were reasons for it, maybe not the best ones, but I hope we all learn from it.
I'll try to learn from it. Maybe I should ask better questions, maybe be more sincere, maybe live a better life. Hopefully something works.
In the end I am not happy at all about how my responses are giving off the impression I don't respect OP's thoughts, which I do, but I just miscommunicated here.
I appreciate the kind remark, I was on the fence about it. My internal dialog was something like: "ok, if I can do an ELI3 that doesn't exaggerate, slander anyone, or wildly oversimplify a nuanced topic, I'll hit it".
I think it met that bar. It's a squeaker, and I certainly won't vigorously argue with anyone who objects that such simplifications inevitably miss important stuff, that's always a reasonable objection.
But this turns out to be a really old story: founders got rich, investors got rich, acquiring entities got bent over a barrel, and everyone yawned until they didn't.
Apologize to HN: Sorry for Trolling
2 points by benreesman on April 17, 2009 | hide | past | favorite | 2 comments
I was reading my comment history and came to the horrible conclusion that I am a troll. In particular I'm the worst kind of troll, the kind where it's not always clear that I'm trolling.
My childlike enthusiasm for debate (or maybe just my lousy social skills) have led me to say things that are nasty, inflammatory, sometimes clearly false, and ultimately that I have no desire to see appear next to my name.
So if I've offended any of you then I'm really sorry. I promise to reform, effective immediately.
I should have said “attempt to reform”, it’s one of the many follies of youth that habits change in a day.
It’s actually pretty recently that I’ve really started to recognize the gravity of this forum, how much scope there is for causing harm or precipitating healthy change: HN is in a sense a sort of “swing state” in the conversation. It’s a small community in numbers on a national or global scale, but is overrepresented in influential posts in technology, the sciences, and the generally “logic motivated” posts in society.
I really am sorry that it’s been such an uneven outcome for so many of the intervening 15 years since I wrote that. I should have started forming a coherent worldview, analyzing its ethical implications, and attempting to master my own worst instincts in the sense of translating that into a compelling argument and actionable “policy platform” in terms of how I write on the Internet.
And in a sense I really owe a debt of gratitude in a perverse way to a set of people and institutions that represent such a clear and present danger to the legacy of the Enlightenment to galvanize me into beginning to get organized around what I believe in and expressing that effectively.
I’m still an awfully small fish to merit a 16-year post history in a kind of tinker-toy opposition research commensurate with my tinker-toy influence.
That’s a lot of effort even with modern automation for a guy who is basically a nobody with an opinion.
Absent automation it wouldn’t be worth anyone’s time, and in either case, I’m really curious to know what induced you to spend yours on that?
Kind of a nothing burger announcement for me. Doesn't seem to be anything here that would noticeably change how I'm approaching my fine-tuning projects.
Friendly notice for anyone optimising their openai token usage: multi_tool use wastes 200 tokens on every call with function calling. You're better off implementing function calling yourself than using their API implementation.
Btw, if you've tried fine-tuning OpenAI models before January and came away unimpressed with the quality of the finished model, it's worth trying again. They made some unannounced changes in the last few months that make the fine-tuned models much stronger.
That said, we've found that Mixtral fine-tunes still typically outperform GPT-3.5 fine tunes, and are far cheaper to serve. This next part is a bit of a plug, but I honestly think we have the simplest platform to fine-tune multiple models (both API-based like OpenAI as well as open source alternatives) side by side and compare quality. https://openpipe.ai
43 comments
[ 4.7 ms ] story [ 77.1 ms ] threadIt's the sound of another opencore AI startup unknowingly participating in NVDA arbitrage becoming irrelevant overnight.
There's some exact words shenanigans here. Indeed may have reduced the number of tokens in the prompt by 80%, but they didn't reduce the cost by 80%: the prompt cost of inferring from a fine-tuned GPT-3.5-turbo ($3.00 / 1M tokens) is 6x the prompt cost of inferring from the base GPT-3.5-turbo ($0.50 / 1M tokens). If prompt tokens are cut to 20% of normal, then that implies the overall cost of the prompt tokens is 120% relative to their normal prompt: a cost increase! That's not even getting into the 4x cost of the completion tokens for a finetuned model.
Of course, Indeed likely has an enterprise contract reducing costs further.
My question is will it be before or after US elections?
The truth is that foundation model companies, while in the limelight now, are probably the smallest slice of the pie, with cloud providers and end user applications going to take the lion share of the profit, and GPU manufacturers in line next. Foundation model companies got hyped because every VC wanted to invest in the company that might monopolize AGI, but in reality we're going to have a landscape dominated by fine tuned open source models in a few years, with closed source models being used for certain niches but generally not worth the cost.
*Under certain conditions
Regarding the monthly scale mentioned in article–we are way beyond that now.
A lot of really smart people worked on this and it was fun to watch unfold.
Google & Cohere give same pricing for non-tuned models vs tuned models.
OpenAIs fine tuning offering is non competitive.
I hope they will adjust their pricing
Given the disproportionate amount of lawyers in any given legislature, never mind the make-up of judges, I can only imagine that a legal stranglehold insulating lawyers from the threat of AI will be one of, if not the, first bit of AI protectionist legislation.
You can't retry a case if you had a bad lawyer, only if procedural mistakes were made.
I just wish they let you make new accounts after deleting an old account, apparently it's something to do with the database when I ask them about it, which suggests that they store data about users even after someone deletes their account.
The CEO was (up to a credible consensus of primary sources) fired for self-dealing in borderline criminal fraud-level ways. I’m not aware of anyone disputing this with any vigor. Then that CEO was fired again for more vague but thematically consistent allegations of flagrantly profiteering off an ostensible charity actively lobbying for de-facto regulator status. Then that CEO made a token concession by disposing of yet another in this cavalcade of personal financial gain presented as urgent imperative.
And the most moderating influence is a guy who has been run out of everything from politics to finance to prestigious academic posts over scandal after scandal in print for 35 years running, and has yet to issue an apology I’m aware of.
But the really high thread-count noose is that Opus is mopping the floor with GPT-4-1106 and 0125, on money that would fall out of Nadella’s couch cushions. And it’s dramatically operator-aligned.
You can’t be a society-scale criminal, keep it up for over a decade, and get lapped by vastly less-resourced competitors in the Valley. You shouldn’t be able to do any of that and still wield any power.
This new breed of tech titans have forgotten something their predecessors knew: it took Wall Street a century to utterly capture its regulators.
Tech people who go big enough? Holmes is serving time, SBF is serving time, they’re not the first.
This isn’t about corrupting the competitive landscape anymore geniuses: this is about staying out of prison.
True. I only hop on GPT-4 when I finish my quota for Claude. I much prefer Claude's long form writing.
https://www.newyorker.com/magazine/2016/10/10/sam-altmans-ma...
https://en.m.wikipedia.org/wiki/Lawrence_Summers
https://en.m.wikipedia.org/wiki/Summers_memo
https://www.pbs.org/wgbh/frontline/documentary/warning/
https://en.m.wikipedia.org/wiki/Loopt
https://en.m.wikipedia.org/wiki/Socialcam
https://www.wsj.com/tech/sec-investigating-whether-openai-in...
https://www.businessinsider.com/sam-altman-hype-openai-silic...
https://archive.is/2023.12.02-101904/https://www.washingtonp...
https://www.reuters.com/technology/openai-ceo-sam-altman-ste...
https://qz.com/sam-altman-openai-fund-1851380019
https://fortune.com/2023/09/20/instacart-ipo-stock-shares-ce...
Altman/Loopt, Siebel/Socialcam, Emmett/Twitch failed and got rich. The people who trusted them lost a lot. People still say they’re smart.
Larry said we should put as much poison in Africa as we can. He wrote it down. He tried to get someone fired, by ratting out his friends. He made bankers angry. They never forgave him. So he moved. Also, Brooksley was going to save America. Larry said she was a woman and be quiet. Also stealing is ok.
Bankers don’t go to prison. They can do anything. But it’s not because they’re rich. It’s because they know important people. Also rich.
Creepy tech people aren’t like bankers. They don’t get in trouble much. But if they’re bad enough, they go to jail.
Someone did the worst thing in a long time. They’ve done it a lot.
He should say sorry and get a different job, because jail sucks.
Or do you think it has to be a TV show, maybe like "Succession" - do you think you'd be able to write something like "Succession"?
I was specifically calculating the 150 - 250 pages to possibly contain enough detail to have many of the explanations that I would have energy to read.
Unfortunately, I'm not going to get this book, so I will have to deal with my everyday life instead.
I was kind of hoping that mentioning "Succession" due to its nature would give it away, that I'm respecting the intelligence of the criticism, because I feel like "Succession" is a really intelligent and insightful show.
Obviously the way I commented it, it didn't come off like that and I was trying to be unnecessarily funny when I shouldn't have been, but I don't 100% regret it, I did it, there were reasons for it, maybe not the best ones, but I hope we all learn from it.
I'll try to learn from it. Maybe I should ask better questions, maybe be more sincere, maybe live a better life. Hopefully something works.
In the end I am not happy at all about how my responses are giving off the impression I don't respect OP's thoughts, which I do, but I just miscommunicated here.
I think it met that bar. It's a squeaker, and I certainly won't vigorously argue with anyone who objects that such simplifications inevitably miss important stuff, that's always a reasonable objection.
But this turns out to be a really old story: founders got rich, investors got rich, acquiring entities got bent over a barrel, and everyone yawned until they didn't.
https://news.ycombinator.com/item?id=567736
Apologize to HN: Sorry for Trolling 2 points by benreesman on April 17, 2009 | hide | past | favorite | 2 comments
I was reading my comment history and came to the horrible conclusion that I am a troll. In particular I'm the worst kind of troll, the kind where it's not always clear that I'm trolling. My childlike enthusiasm for debate (or maybe just my lousy social skills) have led me to say things that are nasty, inflammatory, sometimes clearly false, and ultimately that I have no desire to see appear next to my name.
So if I've offended any of you then I'm really sorry. I promise to reform, effective immediately.
Regards, Ben
It’s actually pretty recently that I’ve really started to recognize the gravity of this forum, how much scope there is for causing harm or precipitating healthy change: HN is in a sense a sort of “swing state” in the conversation. It’s a small community in numbers on a national or global scale, but is overrepresented in influential posts in technology, the sciences, and the generally “logic motivated” posts in society.
I really am sorry that it’s been such an uneven outcome for so many of the intervening 15 years since I wrote that. I should have started forming a coherent worldview, analyzing its ethical implications, and attempting to master my own worst instincts in the sense of translating that into a compelling argument and actionable “policy platform” in terms of how I write on the Internet.
And in a sense I really owe a debt of gratitude in a perverse way to a set of people and institutions that represent such a clear and present danger to the legacy of the Enlightenment to galvanize me into beginning to get organized around what I believe in and expressing that effectively.
I’m still an awfully small fish to merit a 16-year post history in a kind of tinker-toy opposition research commensurate with my tinker-toy influence.
That’s a lot of effort even with modern automation for a guy who is basically a nobody with an opinion.
Absent automation it wouldn’t be worth anyone’s time, and in either case, I’m really curious to know what induced you to spend yours on that?
That said, we've found that Mixtral fine-tunes still typically outperform GPT-3.5 fine tunes, and are far cheaper to serve. This next part is a bit of a plug, but I honestly think we have the simplest platform to fine-tune multiple models (both API-based like OpenAI as well as open source alternatives) side by side and compare quality. https://openpipe.ai