This rather didn't take long for OAI to create*, I remember people giving opinions and discussions that it won't take too long and that openAI should do it[0], so looks like they were right.
Interesting to see where all this leads us and if other major labs follow suit
Edit: decisions voice looks really interesting as well[1]
Decisions voice isn't a product for anyone else who was confused: it's a canned guide for hooking up a voice model to the decision model browser use thing
Since it is fast and understand images, I wonder if it can play video games. I have a harness setup for the LLM play EA FC but even the fastest LLMs are too slow for it. I need to try this with Decisions API
I genuinely do not understand why anyone would pay OpenAI for this. Running something comparable to Jev is pretty trivial. The whole point of paying for ChatGPT is because OpenAI has a bunch of warehouses that can run a zillion-parameter model.
Running a decision model is way easier and much cheaper. Are they really just trying to capitalize on the hype here? It feels like they really have absolutely zero moat.
If you work for a company that has a 3 to 6 month onboarding period for new vendors and a lifetime commitment to maintain a whole bunch of vendor management horseshit for as long as that relationship exists, it makes a ton of sense.
Add in a bunch of model governance and oversight for anything you train yourself and it’s pretty much a slam dunk deal.
If you're in an enterprise that already has a procurement agreement with OpenAI, this means you don't have to onboard another vendor. Bucket platform strategy.
The response to Jev should be the nail in the coffin over whether or not the AI business is a commodity market.
Out of no where Jev appeared as the next round of the price wars. Jev showed the value of System One models. A fast yes/no/confidence score not only is cheaper but also often all people want. Open source versions flood hugging face and now the big players are giving up a potentially big driver of output tokens to keep customers and race to the bottom price wise.
If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
> If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
Hopefully people will flock to whatever product is making its mission to be commodity and the easiest to replace. Really don't want another free ingress, 100$/TB egress Cloud situation.
If you are a business dealing with with anything remotely sensitive then this is not so easy and you are basically forced to do business with a big player.
It's a problem for the security, legal, and IT teams, but not that much for developers, unless they get really particular about their harness. On the other hand, these are the high dollar value accounts that providers want to keep; but if there's no reason to avoid switching, this market really will feel like a utility market (i.e. it'll be like switching ISPs or cell phone providers - annoying but fungible).
The AI companies want to differentiate and become something more than a commodity, even if it's as critical as a utility is.
Enterprises tend to be your largest customers. Especially when current stock values for tech companies are majority predicated on AI becoming a staple in everyday life.
A lot of places already subscribe to both. Indeed, in large enterprises it isn’t uncommon to simultaneously subscribe to Copilot, Claude, OpenAI, Cursor, AWS Bedrock, Gemini, etc — you might subscribe to different ones for different teams/projects/employees/etc, but often the approval by legal/IT/etc is generic not scoped to whoever is using it right now
If you are charged based on usage, you can “soft switch” between them really quickly.
At my job people discuss almost weekly what the best models are for various price/quality points, with our CTO in particular wanting us to make sure we're being cost-effective in terms of what we're using for a given task (his words are basically "I don't want you to use these tools less, I just want you to use them effectively")
We built our harness around swapping endpoints on demand. We can send to one model, many models, cloud or on prem infra. Certainly, not everyone has, but this is the future imho.
Amen. My employer recently switched Cursor -> Claude, and the whole thing stalled out for around 3 weeks while a BAA (Business Associate Agreement, required for HIPAA compliance) was negotiated and signed.
That's pretty much all enterprise software though, it's pretty much how Microsoft stays in business, and even then if the question of switching is even humoured (the idea of a big company switching away from Office would be inconceivable to many), that'd imply its not sticky enough
My gut is that the market for "ai as a tool" aka Claude Code/Codex/computer use/etc is a significantly bigger one than "models behind the scenes of some service". I've seen people use evals a lot in the latter case and very little in the former case (outside of people's whose job is basically to review the new releases). I haven't personally met anyone with something like "here are a bunch of tickets + a snapshot repo checkout, please try to solve them all" eval approach.
Though honestly I'm also surprised by the hype around Jev from a POV of "wait, are so many people just building on these by using them for classification tasks vs something more multi-step or generative?"
For every production use case I build a eval suite which I use for prompt tuning and model evaluation and config. How else do you establish your model and prompt combination works? How do you upgrade to a newer model or decide on a fallback?
This same suite can simply be run very handoff to switch to a new prod model.
Not only can you switch models easily, but their competition can easily duplicate their product. I'm not sure stickiness has been found yet, but my guess is being more of G-Suite for "intelligence" than being a token hawker.
well, it's a function with the string(string) type signature, so it's bound to be weakly coupled, it's literally stringly typed.
If there's efforts to build vendor lock in, it's going to be in the surrounding api, whether it's streaming responses, or this weird prediction thing, or temperature settings, seeds, stuff like that.
And even then, competitors can copy the interface because interfaces are not copyrightable.
The only way they can get lock in is with the data. Right now it’s like a return to the early Web 2.0 days where everyone had an API. You hook the harness up to Slack, Gmail, GitHub, etc. so it can do work.
And like Web 2.0 I’m sure the day will come where the walled gardens return. All the old SaaS companies are starting to charge for agentic access. Proportionally more of the AI budget will go to them instead of the model providers, who are still stuck in this price war
That shorthand rule of thumb is often applicable, but can lead to risky / incorrect assumptions.
The central distinction isn't simply interface versus implementation. It's functional systems and constraints versus protectable expression - a distinction that can cut through both an interface and its implementation.
copyrighting an interface would break anti-trust, as it would mean no competitors can provide their alternative interface implementing products, on either side of the interface.
It would be like patenting a philips screwdriver or screw.
But OpenAI and Ant can be in more than just the commodity part of the business. They can both make frontier models and sticky products on top of those that have network or other effects that make competition tough.
The question isn't whether the market is big enough. It's whether at a race to the bottom on prices is sustainable and worth what investors expected it to be. The original story of OpenAI and Anthropic was that it was a race to super intelligence and whoever gets there first basically captures all the money in the economy. That kind of investment pitch means missing out not just on big returns but perhaps infinite return on investment. It seemed like it'd be true when the market was young and the cost to compete was prohibitive, but now anyone can compete with open weight models that are cheaper and maybe not the best, but are good enough.
You raise $200B to be a high margin low capex business, not an industrial commodity producer with high capex and margins being set by competitors who can duplicate your product and undercut you on price.
> If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
My approach would be per-user (or per-project) "memory".
I don't mean a tack-on like a RAG / document store with "notes" added to it in the background, but an actual medium- and long-term memory. Something like an extension of the KV cache stored in High Bandwidth Flash (HBF), or a subset of the weights trained "online", similar to LoRA.
This would not be transferable to any other base model, so would be excellent "lock in".
The downside is that the memories likely wouldn't be transferable to new models either, but I can imagine solutions to that too. I.e.: Train an MLP to "translate" from the old memory weight space to the new one.
Try making more measured statements, you exaggerate your point and end up being wrong. Of course this new barely used product feature is not all people want. Of course this minor product feature is not the nail in the coffin of whatever argument, they are just responding fast to a smaller competitor by providing that feature themselves, something that happens thousands of times in business, did Instagram prove it was a commodity when it implemented reels by copying tik tok? Did Uber prove it was a commodity when it started offering food delivery? Doesn't make much sense.
It’s interesting that both companies have fully embraced agentic development with unlimited token budgets and yet still can’t expand beyond the original core products
All of those companies that naive HNers used to say they could code in a weekend literally can be coded in a weekend now, so why haven’t they?
If enterprise is the goal they’re really at the whims of the old school software companies because they own no data of their own. Imagine Google decides to push Gemini one day and now ChatGPT can’t write docs anymore.
If somebody from OpenAI or Anthropic is reading this, one way to make the products stickier would certainly be to not make the desktop apps a complete shitshow.
Maybe they could for example point their amazing models to their GitHub issues, and use their power to actually fix the bugs and myriad of papercuts before solving cancer and world war/peace/whathever the stockholders believe they want.
Idk why Jev was even hyped in the first place. Mostly by social media people who are the type to write inspirational LinkedIn articles, Medium posts and make longwinded Youtube videos about subjects they don't know or care to know much about.
It's always been obvious that small models with a specific purpose will outplay the larger more general models. It becomes a question of "do I want to be able to toss _anything_ at this frontier model, have it handle it all but pay the price" or "do I want to toss _specific things_ at this micro model, have it handle it but not be flexible".
It's the same with agentic RAG/search, things are moving towards much smaller search-specific models rather than tossing RAG chunks at a large frontier LLM. It's like how I can ask a multimodal frontier model to identify bounding boxes for objects in an image...but if I want to do that faster/cheaper/at 60fps then I should be using yolo or similar.
I think that technical people tend to underestimate the importance of convenience. If it were up to hacker news, everyone would be on Linux, everyone would assemble their own laptop and nobody would own an iPhone.
Even for technical problems sometimes having something fast and convenient that doesn't require a ton of fine tuning opens up a lot of doors. Those door might've easily been openable previously with a few days of effort. But the difference between a few days and something that you can set up with a quick account and a prompt is massive
I completely agree. First, there's the split between the ones using AI to write coder and there's the Ai that gets used in software.
Writing code is _hard_ because it's by definition accurate logic math will tons of abstractions that build on each other. Our brains are great at this. Frontier models with lotsa context are too. Let's ignore those though.
There are a lot of problems that are simple. Accuracy is important (e.g. is it nudity?), but it's still simple. The difference between Jev and the general idea of small models you described is that Jev (and a million copycats, Jev being a copycat itself) only answers multiple choice. That's clever; why wax poetic when you can ask a question and just get back the multiple choice, standardized tests love them for a reason.
So really it's:
a) "Big Know it All" Models. Slow and pricey, but they'll handle anything you through at them.
b) "Small Domain-Specific" model that still speak answers. They need some amount of "can form eloquent answers" in there with the specialized knowledge so still have a bit of a floor, but totally make sense.
c) "Memorized multiple choice tests, and can read but not write." Sometimes open-ended answers really are a downside when the answer space is sufficiently constrained, and read-side literacy really
Not nit-picking, but (c) really is more different than just a specialized model. And you're right, lots of problems fit that shape — hell, before AI blue up the world that's _all_ computers could do — and many times ambiguity and hallucinations are drawbacks of open-ended answers. You just need to ensure that "N/A" is always a valid answer, or at least that you get a confidence score.
> If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
That's what I have been saying: OpenAI or Anthropic's moat will not be the models, but how they integrated into enterprise clients and make it super hard for them to switch to a different provider. Kind of like Microsoft Office does it. Or Slack. Or Google Work. None of these products are better in any sense, they just cater to the needs of big enterprise clients in another way (customer relationship, sales, maybe compliance) and make it hard to switch
That https://api.openai.com/v1/decisions endpoint is notable because usually when OpenAI define an endpoint like that it ends up as a defecto standard for other providers.
These are essentially zero-shot classifiers; they don't need to be trained for a specific classification task. You could include some natural language context on the rules for classification and it should get good enough accuracy.
1) you can parallelize the request
2) since a structured output is still just text then every previous property of the structured output effects subsequent ones
At least a year ago they were not that accurate when compared to a good many-shot classifier that (1) has 10k training examples and (2) has the learning capacity to learn from 10k training examples (most fine tuned BERTs don't seem to.)
I'm skeptical of the quality of probability calibration for models if you aren't giving them training data. The issue is that there is a prior distribution that's unique to your specific data and the calibration is really sensitive to that.
they are separate questions about the input. One question can be "is this a complaint?" and another "Should we flag this email to security" and a third "Is the sender a Bush era republican?"
All this is for extraordinarily simple decisions. Real world problems often are a lot more complex requiring highly structured outputs covering many output attributes and substructures, for which a proper structured output with a documented schema is better. I think it will be forgotten soon enough.
Deterministic predictions are sought by those looking to offload their decision responsibility to AI, whether to lower perceived legal risk or otherwise. This is fake risk reduction, i.e. "risk theater".
Unfortunately, a deterministic prediction utterly fails to yield an uncertainty measurement which is critical to have in actual risk reduction. If you want the variance in measurement, it is vital to obtain multiple measurements. This also gives a confidence interval.
Ran my decisions evals (still rudimentary, less than 600 calls (UI component selection, chat charting, tag selection, PKM stuff)) on this via OpenRouter against Jev and Mercury Decide. Jev because it has replaced my mt0 efforts by sheer force of affordability (more importantly, the limits running on a MacBook Neo bring even after vocab pruning and quant insanity) and Mercury Decide because I do like dLLM efforts (and I'd like to use fewer model providers if possible).
Preliminary of course, but seems to be slower than Jev and similar to Mercury Decides latency, though not in growing linearly with the amount of input (346ms p50 and 860ms p95, (Mercury Decide also had some extremes up to 1,3s that were around 800ms today, likely preview related)), less "confidence" concerning my ambiguous UI component and response shape specific tasks (have very specific use cases for these models which Luna often fails to meet at 0.6 and lower), lead to a few failed calls which neither competitor had (4 vs 0 for both) and measured more expensive than Jev to boot by a factor of 3,1 times on average (Mercury Decide pricing I think is still unknown).
Basically slower, more expensive and less capable than Jev, roughly on par with Mercury Decide (provided, in my insane set of use cases and requirements that are a PKM focused Firefox fork with multiple infinite canvas using decision models to improve information synthesis from multiple source).
Seems a bit undercooked overall and I'd rather frontier-labs don't jump on bandwagons until they can offer something competitive. In fairness, I have yet to test image input, maybe that makes all the difference. Also, again, mine is unlikely to reflect everyones use case, so interested in seeing others results.
I think releasing something like this makes sense even if it's underbaked, it's still very cheap, and if you have existing enterprise OpenAI relationship it's a lot easier to onboard something like this than set up a new Jev contract.
How these models play out is an open question but existing provider contracts and T&C are important for enterprise.
i would be shocked if luna decides is less generally capable than jev. jev has failed to understand any novel domain I've given it. i have found that i use it only when "some data is better than no data"
Very task-dependent of course and mine are unique to say the least, so could see Luna being better in certain domains, even if my measurements have not shown that yet.
Ran every task twice on each model, some were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul, whether to use the provided tag given the title:
1. Title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
Tag: "house hunting"
Decisions API (2 runs): 0.99, 0.99
2. Title: "S&P 500 index: live chart and news (Bloomberg)"
Tag: "investing"
Decisions API (2 runs): 1.0, 1.0
If you have other examples of requests with unexpected outputs, feel free to email me at by@openai.com and we can try to get to the bottom of it. Thanks for trying out the API!
I realize this is probably too old to see; try without openrouter. Openrouter has caused all sorts of degradations and exotic bugs for us (not just on decision models)
Thanks for the warning. Did a rerun with the official OpenAI API directly and was able to reproduce the issue on those queries, so at least in this case, not OpenRouters fault. Also, shared them with Brian via Mail [0], hopefully useful to them.
Do occasionally cross-compare official with OpenRouter and have yet to encounter anything that wouldn't be replaced by run-to-run variance, but will keep an eye out for it, thanks for the warning. Have, especially with smaller labs, seen OpenRouter not support sometimes critical API features though, Inception Labs being an example. In fairness, the documentation of Inception Labs also had flaws like not noting certain API features being deprecated in their newest model.
Well, Jev doesn't meet any real compliance requirements but OpenAI's models do. So even if Jev is faster, any customers with compliance needs will obviously pick OpenAI because they can't pick Jev out of necessity.
I can run OpenAI on AWS Bedrock and Azure. I cannot run Jev on those cloud providers. For the many companies locked behind cloud providers, we don't have the same options.
They can follow up in N weeks with a better one. Even if your eval is true and it’s worse, planting a flag makes sense. Some people will just use OAI because it’s OAI. No one will remember their week 2 evals in a few months.
> The tulip became a luxury item and many varieties were introduced. The varieties were classified and the most sought-after, prized tulips were the streaked tulips, especially yellow or white streaks on a red or purple background. These flame-like tulips were highly sought after. Interestingly, the streaks or “flames” of the tulip petals were caused by a virus. The virus is the tulip breaking virus, or tulip mosaic virus.
Imagine a worker in a car factory who inspect the car's body.
Pictures from all angles and KI inspection is faster, cheaper more thorough and works 24 hours a day.
And that is only 1 scenario.
Im not questioning that there is value in automated dent testing, im simply questioning how this specialised and limited number of workers.
I think its interesting that this AI functionality couldnt display something that replaces millions of workers and instead presents to us, a useful tool that replaces at best a few hundred of thousand workers.
Is there a saving by automating those workers - of course.
Is that saving worth trillions ?
Big AI is worth more than all the car manuf on earth by several factors, and the only thing they have replaced is a few hundred workers.
Many businesses have classification problems. I worked for a Goverment customer that received a ton of mail.
Workers had to sort things manually because very frequently the form we ask the customer to return (whichhad OMRs) with their documentation wasn't included. So those had to be manually processed by different sub workgroups. There was an entire line of workers whose responsibility was to scan and categorize the mail internally so that it will go to the correct group for further processing.
This entire workload could be replaced by automated scanning machine along with a classification model.
I tried out the multimodality and this is the biggest win, imo.
You can give it a CCTV image (like of a train platform) and ask it to quickly decide actions such as triggering an automated auditory alert, deferring to a larger model for more detailed analysis, etc.
Have there been any signals from Anthropic about matching this? We use AWS bedrock and just switched to Anthropic from OpenAI because of the ZDR guarantee. Would be great to not have to entertain switching back.
214 comments
[ 7.0 ms ] story [ 132 ms ] threadInteresting to see where all this leads us and if other major labs follow suit
Edit: decisions voice looks really interesting as well[1]
[0]: https://news.ycombinator.com/item?id=49802161: OpenAI is well positioned to fast-follow Jev
[1]: https://developers.openai.com/api/docs/guides/decisions-voic...
https://docs.moondream.ai/
but the decisions are stateless... or you plan on passing the last N frames?
Running a decision model is way easier and much cheaper. Are they really just trying to capitalize on the hype here? It feels like they really have absolutely zero moat.
Add in a bunch of model governance and oversight for anything you train yourself and it’s pretty much a slam dunk deal.
Out of no where Jev appeared as the next round of the price wars. Jev showed the value of System One models. A fast yes/no/confidence score not only is cheaper but also often all people want. Open source versions flood hugging face and now the big players are giving up a potentially big driver of output tokens to keep customers and race to the bottom price wise.
If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
Hopefully people will flock to whatever product is making its mission to be commodity and the easiest to replace. Really don't want another free ingress, 100$/TB egress Cloud situation.
The AI companies want to differentiate and become something more than a commodity, even if it's as critical as a utility is.
If you are charged based on usage, you can “soft switch” between them really quickly.
and they ask dumb followup questions after 7 business days when you want different access
They even negotiate with all three aggressively.
So not really.
Though honestly I'm also surprised by the hype around Jev from a POV of "wait, are so many people just building on these by using them for classification tasks vs something more multi-step or generative?"
This same suite can simply be run very handoff to switch to a new prod model.
If there's efforts to build vendor lock in, it's going to be in the surrounding api, whether it's streaming responses, or this weird prediction thing, or temperature settings, seeds, stuff like that.
And even then, competitors can copy the interface because interfaces are not copyrightable.
And like Web 2.0 I’m sure the day will come where the walled gardens return. All the old SaaS companies are starting to charge for agentic access. Proportionally more of the AI budget will go to them instead of the model providers, who are still stuck in this price war
That shorthand rule of thumb is often applicable, but can lead to risky / incorrect assumptions.
The central distinction isn't simply interface versus implementation. It's functional systems and constraints versus protectable expression - a distinction that can cut through both an interface and its implementation.
It would be like patenting a philips screwdriver or screw.
You raise $200B to be a high margin low capex business, not an industrial commodity producer with high capex and margins being set by competitors who can duplicate your product and undercut you on price.
My approach would be per-user (or per-project) "memory".
I don't mean a tack-on like a RAG / document store with "notes" added to it in the background, but an actual medium- and long-term memory. Something like an extension of the KV cache stored in High Bandwidth Flash (HBF), or a subset of the weights trained "online", similar to LoRA.
This would not be transferable to any other base model, so would be excellent "lock in".
The downside is that the memories likely wouldn't be transferable to new models either, but I can imagine solutions to that too. I.e.: Train an MLP to "translate" from the old memory weight space to the new one.
> is .. all people want.
Try making more measured statements, you exaggerate your point and end up being wrong. Of course this new barely used product feature is not all people want. Of course this minor product feature is not the nail in the coffin of whatever argument, they are just responding fast to a smaller competitor by providing that feature themselves, something that happens thousands of times in business, did Instagram prove it was a commodity when it implemented reels by copying tik tok? Did Uber prove it was a commodity when it started offering food delivery? Doesn't make much sense.
All of those companies that naive HNers used to say they could code in a weekend literally can be coded in a weekend now, so why haven’t they?
If enterprise is the goal they’re really at the whims of the old school software companies because they own no data of their own. Imagine Google decides to push Gemini one day and now ChatGPT can’t write docs anymore.
Maybe they could for example point their amazing models to their GitHub issues, and use their power to actually fix the bugs and myriad of papercuts before solving cancer and world war/peace/whathever the stockholders believe they want.
It's always been obvious that small models with a specific purpose will outplay the larger more general models. It becomes a question of "do I want to be able to toss _anything_ at this frontier model, have it handle it all but pay the price" or "do I want to toss _specific things_ at this micro model, have it handle it but not be flexible".
It's the same with agentic RAG/search, things are moving towards much smaller search-specific models rather than tossing RAG chunks at a large frontier LLM. It's like how I can ask a multimodal frontier model to identify bounding boxes for objects in an image...but if I want to do that faster/cheaper/at 60fps then I should be using yolo or similar.
Even for technical problems sometimes having something fast and convenient that doesn't require a ton of fine tuning opens up a lot of doors. Those door might've easily been openable previously with a few days of effort. But the difference between a few days and something that you can set up with a quick account and a prompt is massive
Writing code is _hard_ because it's by definition accurate logic math will tons of abstractions that build on each other. Our brains are great at this. Frontier models with lotsa context are too. Let's ignore those though.
There are a lot of problems that are simple. Accuracy is important (e.g. is it nudity?), but it's still simple. The difference between Jev and the general idea of small models you described is that Jev (and a million copycats, Jev being a copycat itself) only answers multiple choice. That's clever; why wax poetic when you can ask a question and just get back the multiple choice, standardized tests love them for a reason.
So really it's: a) "Big Know it All" Models. Slow and pricey, but they'll handle anything you through at them. b) "Small Domain-Specific" model that still speak answers. They need some amount of "can form eloquent answers" in there with the specialized knowledge so still have a bit of a floor, but totally make sense. c) "Memorized multiple choice tests, and can read but not write." Sometimes open-ended answers really are a downside when the answer space is sufficiently constrained, and read-side literacy really
Not nit-picking, but (c) really is more different than just a specialized model. And you're right, lots of problems fit that shape — hell, before AI blue up the world that's _all_ computers could do — and many times ambiguity and hallucinations are drawbacks of open-ended answers. You just need to ensure that "N/A" is always a valid answer, or at least that you get a confidence score.
While the internet will live on, many of the companies that were initially leaders in it will not.
I’d bet by 2039 either Anthropic or OpenAI will have been bought by SpaceX. One will be the Sun Microsystems of this era.
That's what I have been saying: OpenAI or Anthropic's moat will not be the models, but how they integrated into enterprise clients and make it super hard for them to switch to a different provider. Kind of like Microsoft Office does it. Or Slack. Or Google Work. None of these products are better in any sense, they just cater to the needs of big enterprise clients in another way (customer relationship, sales, maybe compliance) and make it hard to switch
I'm skeptical of the quality of probability calibration for models if you aren't giving them training data. The issue is that there is a prior distribution that's unique to your specific data and the calibration is really sensitive to that.
I know it's just a little typo but it made my morning :)
dejure: legal
defecto: enshittified
(adj.): The defecto way to playback music is Spotify.
You can throw a user bio at it, like "NAME: John Smith, AGE: 71, LOCATION: California" and ask OpenAI:
* Is this user located in the United States?
* Is this user located on the East Coast?
* Is this user located on the West Coast?
* Is this user old enough to vote?
* Is this user old enough to retire?
4 out of 5 of those are all going to resolve Yes, with a far greater than 0.5 rate.
Unfortunately, a deterministic prediction utterly fails to yield an uncertainty measurement which is critical to have in actual risk reduction. If you want the variance in measurement, it is vital to obtain multiple measurements. This also gives a confidence interval.
* Pass/Fail evaluation of a assortment of work product againt a given standard. * Document classification without OMRs. Luna can take image inputs.
Preliminary of course, but seems to be slower than Jev and similar to Mercury Decides latency, though not in growing linearly with the amount of input (346ms p50 and 860ms p95, (Mercury Decide also had some extremes up to 1,3s that were around 800ms today, likely preview related)), less "confidence" concerning my ambiguous UI component and response shape specific tasks (have very specific use cases for these models which Luna often fails to meet at 0.6 and lower), lead to a few failed calls which neither competitor had (4 vs 0 for both) and measured more expensive than Jev to boot by a factor of 3,1 times on average (Mercury Decide pricing I think is still unknown).
Basically slower, more expensive and less capable than Jev, roughly on par with Mercury Decide (provided, in my insane set of use cases and requirements that are a PKM focused Firefox fork with multiple infinite canvas using decision models to improve information synthesis from multiple source).
Seems a bit undercooked overall and I'd rather frontier-labs don't jump on bandwagons until they can offer something competitive. In fairness, I have yet to test image input, maybe that makes all the difference. Also, again, mine is unlikely to reflect everyones use case, so interested in seeing others results.
How these models play out is an open question but existing provider contracts and T&C are important for enterprise.
Ran every task twice on each model, some were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul, whether to use the provided tag given the title:
1. > Tile title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
> Tag provided: "house hunting"
> Jev results: 0.69 (yes), 0.72 (yes)
> Luna results: 0.21 (no), 0.21 (no)
Of course, tags can be a bit subjective, but in this case, I'd argue there was an objective correct answer that Jev got which Luna did not.
Also reran Jev [1] and Mercuy Decide [2] (which got 0.83) with same input, for reference.
[0] https://imgur.com/a/nAkCZiG
[1] https://imgur.com/a/scIa9nF
[2] https://imgur.com/a/M7Mm1l7
Do occasionally cross-compare official with OpenRouter and have yet to encounter anything that wouldn't be replaced by run-to-run variance, but will keep an eye out for it, thanks for the warning. Have, especially with smaller labs, seen OpenRouter not support sometimes critical API features though, Inception Labs being an example. In fairness, the documentation of Inception Labs also had flaws like not noting certain API features being deprecated in their newest model.
[0] https://github.com/HominisBrowser/Classifier-Eval
That's a big motivator.
[0] - https://news.ycombinator.com/item?id=49976996
There still is a bit, but not as much.
Will this get folded into models / post training pipelines at some point and make them better at calibrated outputs?
Source: https://www.canr.msu.edu/news/tulip_mania_the_history_of_the...
What's old is new.
I think its interesting that this AI functionality couldnt display something that replaces millions of workers and instead presents to us, a useful tool that replaces at best a few hundred of thousand workers.
Is there a saving by automating those workers - of course.
Is that saving worth trillions ?
Big AI is worth more than all the car manuf on earth by several factors, and the only thing they have replaced is a few hundred workers.
Workers had to sort things manually because very frequently the form we ask the customer to return (whichhad OMRs) with their documentation wasn't included. So those had to be manually processed by different sub workgroups. There was an entire line of workers whose responsibility was to scan and categorize the mail internally so that it will go to the correct group for further processing.
This entire workload could be replaced by automated scanning machine along with a classification model.
Im asking the audience to quantify the value of the jobs that are replaced against the valuation of AI.
Open source classifier models you can run and train locally on CPU
- Cost : It is the same for both scenarios $0.10 per 1M tokens
- Speed : decisions is 10x faster than responses API
- Quality : I guess if we compare with luna which is a pretty good model it itself, both will be at par
So essentially it has to do more with speed vs any other factor.
You can give it a CCTV image (like of a train platform) and ask it to quickly decide actions such as triggering an automated auditory alert, deferring to a larger model for more detailed analysis, etc.
Did you get that reversed? OpenAI has a ZDR guarantee while Anthropic doesn't.
I think Jev had put significant effort here and its not clear if luna will be well calibrated in this way.