Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
I'm trying to organize about 400k photos. What kind of questions are you asking it and what are you doing with that info? Applying tags/labels? Sorting into albums?
Basically just tags, but the questions branch based on the answers. Photo vs Graphic, Portrait vs Landscape, Person/People/No one, etc.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
It’s just the difference between the kind of companies they are. For Google, being a couple of months behind in something like this is completely insignificant. Google doesn’t need to compete on that basis. Their customers for this sort of product are almost all enterprises, who aren’t rushing to use the latest new thing.
I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Yeah, a few years back I briefly worked on the outlines library that did alternate sampling/constrained outputs (classes, json, grammars etc).
It was pretty obvious that calibration was a major problem.
You might ask a question, and say your classes/target tokens are “high”/“medium”/“low”. Well maybe the model really want to say “The” first, and none of your classes might be in the top 1000 tokens for all you know. The models were really not trained to produce one word answer but sentences.
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels.
Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
Unless I am misunderstanding you then I don't think this is correct. Chain of thought was first proposed in 2022. Assuming you mean reasoning models then the correct comparison is GPT-o1, which came out in September 2024, 2 months prior to R1 lite preview. In the next 2 months, prior to the release of the full R1 model, both qwen and gemini released reasoning models. Dario explicitly notes in early 2025 that most major labs had been working on reasoning models in 2024, and explicitly credits OAI for pulling things forward with their release.
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
I think if you were paying attention at the time, CoT and ReAct loops were already "well known". Lots of GPT wrappers at the time were doing it, I remember doing CoT to help measure the calories for a recipe. The problem is tokens then were quite expensive and doing experimentation on it was pretty expensive for an individual. I once accidentally spent $20 trying to categorize a Who's Hiring post from here lol.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
Absolutely. Very much the difference between "people can hack this together themselves" and "all the major AI companies ship models and agents designed around this". Research/experiment vs product.
From what I’ve gathered, it doesn’t work at all like a normal classifier. As in, for a classifier you’d train it for a specific task to reduce the cross entropy. So, in a hand-wavy way “the model knows the truth”. For Jev-type models it’s more like an LLM, in the sense that the “truth” is in the context you provide. But instead of producing a completion it fills in the schema. But it’s always - “given the context, the schema needs to be filled in like this”
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
Dont under estimate the power of branding/positioning/marketing. For more than a year I have been using a RL'd Qwen fine tune optimised for just taking a decision and outputting the decision number, initially I didnt have much training data so after a few manual examples I asked deepseek to make them for me and had codex train it.
I compared it to jev when it first came out and it was mostly neck and neck for my use case.
I can't have been the only one in fact, I'm sure there must have been other people who probably even open sourced, but I couldn't find anything. I think the question going forward will be how does one keep abreast of interesting stuff happening, now that its de-coupled from direct human labour so de-bottlenecked on production capacity - just bound by how residual time/attention one has (Terence Tao was talking about this exact predicament in the latest tyler cowen podcast).
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
> Which makes me wonder: how do these models calculate probabilities?
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for all uses of decision models.
In some sense everything an LLM does is by unprincipled methods. But it works! We judge it empirically post hoc by its performance rather than relying on tuning.
Hell, human thinking is unprincipled in basically the exact same way.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
That also makes me wonder: what other things are actually achieved by now, are up for grabs, and will seem obvious in retrospect, but we don't notice it yet?
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
I have sent to ~22-30 million requests to OpenAI's completions api since early 2023 asking to classify pieces of text. I'm pretty sure many have sent even more.
I think they just didn't have the incentive to offer a cheaper version that does the same thing.
Startup / Founder focus vs Large Organizations with no vision.
With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma 4 26A4. That makes me optimistic for this release, despite the high cost.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
It can be used a fast/cheap router, if a response is above a specified probability then send it to LLM based workflow branch, otherwise nothing more to do
I work in manufacturing and would like to use LLMs for scenarios where there's obviously still rules for what should happen next, but there's just so many variables that I can't just code it into a decision tree easily.
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
2a) decision-tree for *who* to send it to for reworking things
3) scrap
As I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
Task classification - given a set of classes, return the most likely class for each task
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
I still have no idea why this is useful. Models are atrocious without reasoning, for classification and otherwise, and this doesn't even just eschew reasoning afaik, but parts of the models as well for lower latency and cost. I genuinely do not see the point.
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
I think that is exactly the value prop, you don't need to train a classifier. You get something that works well enough to validate your product/feature/whatever before investing in training.
Opaque classifier as a service, for when you want to discriminate people by race and social status assumed from name, but want some other entity to be responsible for plausible deniability of the process.
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
Audio, Video and File input? That is very appealing and I can imagine a few very unique ways to leverage this if it works well across all modalities.
For what it's worth, ran my quick and dirty evals with it and got on average 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team.
141 comments
[ 0.14 ms ] story [ 17.5 ms ] threadHowever, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
It seems unlikely this will be around in 6 months at this rate.
Google is maintaining the AI models and APIs for much longer, even if many people don't know about them unless they use them to build things
Google kills consumer products because most consumers don't pay for them directly, so they often end up not making enough money.
With enterprise products, they have actual paying customers, so the situation is different.
I say plant a thousand flowers and cull the ones that don't bloom.
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
It was pretty obvious that calibration was a major problem.
You might ask a question, and say your classes/target tokens are “high”/“medium”/“low”. Well maybe the model really want to say “The” first, and none of your classes might be in the top 1000 tokens for all you know. The models were really not trained to produce one word answer but sentences.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
Chain of thought was that way. DeepSeek came out, and not long thereafter everyone had adopted chain of thought.
https://darioamodei.com/post/on-deepseek-and-export-controls
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
Decision models are really not much different than that, but (usually) scaled up to bigger causal models.
If you are looking to do Jev-like things with smaller models, check into NLI.
Because it's especially parsimonious with token use.
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
You're leaving out the UX (DX, developer experience) in this.
And the improved zero-shot performance when you don't have a dataset yet (could argue this falls under DX).
Not to mention the faster time-to-decision and order-of-magnitude cheaper costs.
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for all uses of decision models.
Hell, human thinking is unprincipled in basically the exact same way.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
I think they just didn't have the incentive to offer a cheaper version that does the same thing.
So they already knew this type of thing was useful.
bugattis
Getting a response in 250ms opens up a lot possibilities over 750ms.
No I think Jev will do just fine. Not because Google can't even ship this day 1 on their enterprise platform (https://docs.google.com/forms/d/e/1FAIpQLSeupHxao3ajUvl9Dt7l...) but has time for moralizing a decision API.
It's not the spirit of a competitive company.
https://developers.google.com/edge/mediapipe/solutions/decis...
Standard practice by the big G.
They don't need to be the best, they just need to not force a customer to go elsewhere.
I suppose the key question is come 5 years time is enterprise going to just become happy to use openrouter, or will they lose competitiveness/ market share to those who do.
Fyi artificialanalysis currently puts drex highest on price vs quality
https://artificialanalysis.ai/evaluations/artificial-analysi...
Evals. is this hitting the criteria or not? (Test case passes or fails)
https://chromestatus.com/feature/5155080092385280 https://evergreen-labs.org/products/credibility-analyzer
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
3) scrapAs I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
We use it for our critique looping within our own harness.
But decision models can do things like write SQL queries, if you're willing to put a little bit of trickery upfront.
My team's writing a paper about it for ICDE 2027 as we speak (as a follow-up to the DICE demo/paper from VLDB 2014).
But my Linkedin post[1] has a quick summary of the idea, without explaining everything we do there.
[1] - https://lnkd.in/p/gAsWcrCd
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
It comes with a bunch of pre trained classifiers, and you can use the tooling to train your own
There are also integrations with workflow tools like n8n
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
Apparently the two teams aren't talking to each other.
What differentiates this from training a classifier model? That it comes pretrained with general knowledge?
Hey Jev, what mortgage rate multiplier should a person named James Smith get?
For what it's worth, ran my quick and dirty evals with it and got on average 4x lower brier over Jev (2x - 6x depending on task), which is an immense improvement and the first time a model so significantly has outperformed Jev in my experience. p50 was also over 100ms lower though all this cost twice as much across the entire eval end-to-end.
Dare I say, this is a very impressive release by the Gemini team.