Can someone explain how so many folks managed to build decision models within days or weeks after Typesafe came out with Jev? Is this concept of decision models been in the works for a while? Is it easy to copy?
Transformers output a set of probabilities over outputs. For ChatGPT etc, those are predictions of what the next token will be. But it can also be a structured list of options or classes. Jev mostly innovated on the interface, API, and product concept around this, and made it click for a large number of people. Unfortunately for Jev, it's very easy to copy an API, and any pretrained LLM can be adapted to work in this way.
You just have to fine tune an LLM like Qwen on some synthetic data to do so. There was even someone that had a model that was exactly like Typesafe and published their work a year before Jev (but wasn't marketed as heavily since it was academic).
Jev created accessible/programmatic ergonomics around a general purpose classifiers, and did it very well; ie intuitive api and structured data approach.
Anyone can copy that and apply to an array of models - stripped down LLMs or already slim/highly performant traditional classification architectures (just wrap inference with an api that inputs/outputs the same structured data).
Jev, I think, would say their advantage is the intelligence of their models and training data including calibration: https://medium.com/code-applied/calibrated-classifiers-makin... (which i still struggle with in the general application... there's no free lunch with these things).
What's new is "smart" decision models than you can supposedly use on anything without additional training.
If you have a very narrow use case you can train a BERT based decision model on a laptop an hour if you have good data to train it on. It'll answer faster than the roundtrip to clef/jev and use <1gb memory
They are not too difficult to train if you already have infra to train regular LLMs. You can typically replace a few layers train them alone and you're off to the races.
Getting training data that works well for calibrated classification objectives is difficult.
I hear conflicting opinions (including my own) about how well calibrated each of these are. Jev seems to be the best.
But the jev release made obvious the PMF for these models, and the underlying reality is that calibration really doesn't matter much when you're replacing usecases where people were using damn LM head softmax probabilities before, which are nowhere near calibrated.
So now everyone simply finetunes qwen and makes a compared-to-regular-LLM vastly cheaper decision model. And it works for majority of usecases. People mostly only care about accuracy, not confidence.
It's not a new concept, it just took someone adding on to the approach and refining it. I never deep dove it, but I assume JEV is sort of like how Sora works? They had a blog post about how it has a sort of tiny LLM, which OpenAI's small LLMs are insanely good and well defined. I think any lab tackling this with a from-scratch model could yield affordable alternatives that are highly competitive.
It seems insanely obvious at least to me, that JEV is the new hot thing for the AI field since they give you stronger output that isn't... flat out wrong, that alone is impressive.
Most answers explain the LLM-based approach to these models, which is also what Typesafe did with Jev. However, depending on what you need, there are far simpler classification models, and for a lot of use cases, these models can be way faster and more accurate than Jev
But, for these adhoc models, you need to understand the task more, collect some data and train the model (on CPU, no need for GPU). So Jev-like models are a great way of getting a hosted general decision model, but if you have a very narrow task or set of tasks, you might be better off with some more basic models that you can run on the same server you run other things or even on your laptop
Wow, Cloudflare is definitely buying some goodwill from me. Just consistently interesting new releases alongside and solid products at great prices.
Seems nearly too good to be true.
That's probably driven by their own internal need to show the model images of emails and webpages to detect phishing, despite obfuscation of the underlying HTML.
What even are these new "decision models?" Take an existing LLM, feed it a prompt, force it to pick a choice; decode is 1 token (or rather, the whole logit set for only that last token; token implies selecting one logit) so you made a choice. That's it?
Yes but optimized specifically for the purpose. Using that for "decision making" is also not a new use case, but turned out to be new to many people. Which is great, I hope they make something cool with it!
The basics are pretty simple. And depending on what your specific need is, the model can be really really basic, fast and super effective (ie. run on a mobile device and process thousands of requests in <100ms)
I've been playing with this for the last year or so. Started with a personal email classifier, also did benchmarks with some public datasets, then created a couple classifiers that could play Doom, and now I've been trying out some other experiments, like a request proxy/router to automatically choose a classifier and fallback to LLM to handle unseen requests
Jev did a great job at creating hype, but also at shaping the concept and space of "decision engine" or "decision model". People were already doing this with LLMs, which is very inefficient for most tasks like that, and the Jev guys figured there was a market there. It seems like they were right, and now there's a rush to flood the space, taking advantage of the hype window
In general, training a general purpose classifier is something lots of people have worked on for a long time. Large Transformer models themselves are typically "generalists" already, so structured generation and constrained decoding have given you the ability to use an LLM as a general classifier for years. It's an incredibly common pattern for working with LLM judges or any sort of branched decision making workflow.
A lot of people who are a bit less familiar with the field saw the hype around Jev and presumed that the reason it was so exciting was that it was a fundamentally new interface for working with an LLM. And that additional excitement drove even more attention to Jev. But fundamentally, TypeSafe's announcement was that they found a particular architecture/training paradigm that resulted in a model for this particular interface that had incredible accuracy, very low latency, and for which they could offer inference at a super low cost.
I've not kept up with the flood of Jev clones that have been released, but I think this is just typical for any new component in deep learning that gets popular. There are an absurd number of open source autoregressive LLMs and fine tunes you can use. The thing that makes one more popular than the other is typically the general performance of the individual model.
But training a model for this purpose, or emulating the procedures described in Jev's papers, isn't something that would be beyond the capabilities of any lab. It's not an entirely alien architecture or approach.
The bigger question for TypeSafe as a company would be if other teams are producing Jev-like models that win on performance or cost. Like I said, I haven't followed the reports super closely, so no idea if that's the case or not.
> The bigger question for TypeSafe as a company would be if other teams are producing Jev-like models that win on performance or cost. Like I said, I haven't followed the reports super closely, so no idea if that's the case or not.
If CF's benchmark is representative and sufficient, Clef outperforms Jev!
Models by themselves don't guarantee market capture. Rather, its how they integrate. I think a lot of folks are burned by the closed nature of many models.
Yes, it's easy. The thing people are missing (especially those believing AI is a "dead end" and "not transformative") is that the field has been advancing so fast in the past few years, that there's lots of such unexplored avenues, unpicked low-hanging fruits, that everyone just raced past. We've barely begun exploring the capabilities ML brought us - patterns, applications, and architectures.
Now that we're hitting against the hardware supply limits of global economy, I expect more people to go back and revisit the things left along the way in the mad rush to "just throw more compute at it / make a bigger model" - and thus many more cases like Jev to show up in the next few years.
Yes it's easy for an established shop, all they need to do is to tweak the post-training workflow. "Decision model" is the same kind of marketing as "LRM" attempted by OpenAI when RL CoT was new (to hyped up crowd). It's still fundamentally a classifier used for "decision making", games and RP were using generalist models and constrained outputs to do what the DOOM demo does for years.
The interesting part is also the easy part. The model and architecture are not hard for an experienced machine learning engineer to build.
The hard part is the data and evaluation. Sure, it’s not that hard to build a fast model with good predictive power. But fast at doing what? You probably don’t care about classifying whether a hotdog is a sandwich (which is the Jev demo).
Yeah, these AI companies have some weird paradox that if they actually had a model that was super efficient and could arbitrage cost/intelligence of other inferior models, they would keep everything about it secret. If an intelligence research group had something groundbreaking, they would just dump their own money into the magic money machine.
Instead, to make up for the lack of economic viability of their models, they are forced to release publicly to get marketing to get others to pay based on hype.
I wonder if a good usecase for this would be cloudflare's WAF rules. Give broader request context to the decider and let it pick type of challenge/block traffic directly.
Am I hearing this right, that they made a decision model based on Typesafe's new paradigm, and actually made a model better than Jev based on Typesafe's own ranking?
It's not a "new paradigm", it's a low-hanging fruit that's been lying around for years; Typesafe were the first to bother to stop and pick it up, and market the shit out of it. But it was still a low-hanging fruit.
There are many, many of those left around, because AI frontier is moving forward so fast, everyone is racing ahead. Which is why I laugh when people say AI is not transformative and LLMs are a dead end (and my favorite, "what are we going to do with all those GPUs when the bubble pops?"). Even if SOTA LLMs hit a hard capability limit tomorrow and never advanced again, there's a good decade of growth and advancement to be extracted just from all the low-hanging fruits that were left unpicked along the way.
Synthetic data is key here! Compared to 5 years ago, we now have oracles that can generate massive data sets of perfectly labeled multimodal data, practically for free. The number of architectures that can benefit from that is innumerable, and far beyond just LLMs themselves. On top of this, LLMs can implement any architectural ideas you have, and write custom tools to manage training and evaluation.
Whether or not LLMs can self-improve their frontier capabilities, they can absolutely create a wake for themselves that accelerates everything else that's training on their synthetic data. We'll see every architecture of the past 40 years suddenly show leaps and bounds.
Agreed on the directions, it might appear to be piece of cake, but the "practically for free" part hides the hard bit: realism. Try simulating flight-search data with source, destination, flight numbers, routes, schedules , try a diff it against a real corpus. Even frontier models produce data that looks right at first sight, but breaks on joint distributions (a flight number that never flies that route, layovers that violate minimum connect times). last mile is the daunting task.
To lean into your example: if you want to design travel search, you might envision a system that goes from a user's typed query -> a structured query -> executed against a database of real-time flights -> judging the result for feasibility of the resulting proposals created by those queries.
What you can do with LLMs is reverse this: from any numbers of snapshots of flight data, you can create large numbers of plausible user queries, based on your data, that are known to be feasible or infeasible. And now you have a labeled data set to train a model that focuses solely on the query-creation and judgment systems. And you can experiment with whether having a more flexible query protocol leads to higher success rates without sacrificing accuracy, or whether you can generate that last-mile feasibility check as a combination of auditable code checks alongside AI-based judgment.
LLMs don't absolve you of having to break down your system architectures into components that have well-defined boundaries (though certainly they can help with that design). They do make those components feasible to solve at scale.
Yup, one output token, and read the logprobs of the posible tokens that could have been thos 1st token. Plenty of systems already do this; Typesafe's fundraising and marketing just made it visible.
Good to see interest broadening beyond "just extend thinking." More approaches in the toolbox means fewer problems get treated as nails.
> what are we going to do with all those GPUs when the bubble pops?
Your comment here made me laugh, because I think we will finally be able to play Crysis at 10fps.
Just kidding, of course. GPU half lives are quite a bit less than standard compute half lives, no?[0] That's what I've been trying to understand regarding data centers focusing as GPU clusters -- seems like the ROI window would have to be very short for the capitalization.
I'm all for rebranding discriminative models as decision models, though.
"Discriminative" always had pointlessly bad optics, but I knew it was over when I started seeing prominent machine learning researchers who p=100% knew better describe discriminative models as generative because that was the buzzword of the year. "Decision model" sells the value proposition much better and doesn't sound like an anti-woke crusade.
Decision models have long been called `choice` models to may understanding. Huge statistical literature on dynamic discrete choice models and inference!
A lot of people claim to have made better than jev, there's a jev benchmark, I have tried many of those models and they eventually end up failing, a non trivial task which doesn't seem like much but reminds me of the svg pelican bench is games, have one of these decision/classifier models play a game, hook it up to the input, most of the ones that are supposedly on jev level end up playing a terrible game, showing that they are very narrow. Cloudflare doesn't compare to the top open bench alternatives, I just finished downloading it and will compare it to jev for non trivial tasks tonight.
Public benchmarks are easy to cheat, if I am typesafe I would also release a public benchmark to distract otherwise competent people in overfitting to a benchmark instead of making something actually useful.
Diogo very much is against public benchmarks ;)
You take Qwen3.6 35b on a 5090rtx, and here you get a higher score than Jev, for 2sec more latency on average.
So yeah it's not subsecond, but I am sure that if I had VC money, I could too get within 500ms too!
"Clef builds upon this concept, but uses a different base model as the backbone. We currently use Qwen as the base model and post-trained it to suit decision model use cases. "
There was some parallel independent work from Sep 28. They mention it in the blog post. But I am sure Jev has opened a lot of eyes on the possibilities.
"In the same week that Jev came out, we posted about some experiments [1] we had with our own homegrown decision model."
From their blog post, it sounds like it is still 10x as slow as Jev.
As far as I've seen, all of the Jev-compatible projects simply take an LLM, hack off a layer or two at the end, and call it good. Some of them spend more work than others trying to back-estimate in accurate probabilities.
Yeah I think people were already using either of these as part of traditional software workflows):
A) Structured outputs from LLMs (doesn't need fine tuning but can be expensive)
B) Classification output from fine-tuned BERT-like or GLiNER models (is calibrated well and has cheap/fast inference)
What Jev did is combine the advantages of both A and B into one model/product, and create a really good API.
They claim that a key innovation is how they've trained the model using what they call RLCD (RL from calibrated decisions). So it's not just that you can get the outputs (which is easy to add to any LLM) but that the different primitives they expose (Choice, Score, Noul) have each been calibrated. For example, they claim that if you use the Score primitive (which gives you probabilities along a bunch of choices representing a continuum) that's not just using the more general 'Choice' primitive under the hood. It's been calibrated separately.
I don't know how many of the Jev-like things we've seen do that. For example Cloudflare offers a Jev-like model with the same API, and which they say was trained with RLCD. But I don't know whether Choice and Score are different under the hood, or whether Score is just sugar on top of Choice. (Should be easy to test this, but I haven't done it.)
You make it sound like this is some noteworthy timeframe. I'd be more surprised if Jev wasn't immediately made obsolete within days of release (which was effectively the case), especially when backed by a company like Cloudflare lol
ML moves quick. Add the extreme hype and cash floating around in this space and you can expect that anything resembling something novel and relatively untapped is going to be pounced on and turned over basically immediately.
Traditional LLMs doing structured output is like trying to fit a square peg in a round hole. It's just not the right tool for the job. They can do structured output, but it's janky.
With Jev, structured output is its native format. Jev is just way faster, cheaper, and outright better for a lot of things.
Note that while all of this is great, this is nowhere near a sort of "ChatGPT moment". It's a cool new thing, and it's way better at certain tasks, which could be big. That's all though.
The weights have permissive licensing, but the data and training pipeline are not published to reproduce them from their proprietary Qwen starting points. Weights are not "source."
With all these new Jev-like models popping up, has anyone actually started building anything with them yet? It's odd how quickly they've multiplied despite being relatively niche in their use cases, as far as I can tell. I suppose they're simple and cheap enough to make that it's a sort of 'why not' thing for a lot of these companies.
I'm trialling using it to scan user signups for signs of gambling spam, phishing etc. My current process uses an LLM, but seems like a good fit for these classifiers.
Is there a Jev-like model I can run on my Mac? Something like Ollama? Or what's the best way to play with it? Is there a cheap/free API service eg on OpenRouter?
168 comments
[ 0.25 ms ] story [ 52.6 ms ] threadAnyone can copy that and apply to an array of models - stripped down LLMs or already slim/highly performant traditional classification architectures (just wrap inference with an api that inputs/outputs the same structured data).
Jev, I think, would say their advantage is the intelligence of their models and training data including calibration: https://medium.com/code-applied/calibrated-classifiers-makin... (which i still struggle with in the general application... there's no free lunch with these things).
If you have a very narrow use case you can train a BERT based decision model on a laptop an hour if you have good data to train it on. It'll answer faster than the roundtrip to clef/jev and use <1gb memory
Getting training data that works well for calibrated classification objectives is difficult.
I hear conflicting opinions (including my own) about how well calibrated each of these are. Jev seems to be the best.
But the jev release made obvious the PMF for these models, and the underlying reality is that calibration really doesn't matter much when you're replacing usecases where people were using damn LM head softmax probabilities before, which are nowhere near calibrated.
So now everyone simply finetunes qwen and makes a compared-to-regular-LLM vastly cheaper decision model. And it works for majority of usecases. People mostly only care about accuracy, not confidence.
Many people seem to have run into the same question and started working out the answer.
It seems insanely obvious at least to me, that JEV is the new hot thing for the AI field since they give you stronger output that isn't... flat out wrong, that alone is impressive.
But, for these adhoc models, you need to understand the task more, collect some data and train the model (on CPU, no need for GPU). So Jev-like models are a great way of getting a hosted general decision model, but if you have a very narrow task or set of tasks, you might be better off with some more basic models that you can run on the same server you run other things or even on your laptop
Or are companies/people already building this based on say an arXiv docs? n
---
The pricing is ... hm more expensive but not at the point I won't give it a try due to the embeded vision encoding
Latency won't be that good, but could still work similarly. Simply force the structured output of a LLM to the given schema.
Probably also easy to train because we can use stronget LLMs to generate input/output data, or even synthetic data is easy to generate.
It's not really a new technology, it's more like a new use-case.
https://github.com/blockbrain-ai/cygnet-recipe
https://www.youtube.com/watch?v=AzxoU7kxjig
I've been playing with this for the last year or so. Started with a personal email classifier, also did benchmarks with some public datasets, then created a couple classifiers that could play Doom, and now I've been trying out some other experiments, like a request proxy/router to automatically choose a classifier and fallback to LLM to handle unseen requests
Jev did a great job at creating hype, but also at shaping the concept and space of "decision engine" or "decision model". People were already doing this with LLMs, which is very inefficient for most tasks like that, and the Jev guys figured there was a market there. It seems like they were right, and now there's a rush to flood the space, taking advantage of the hype window
In general, training a general purpose classifier is something lots of people have worked on for a long time. Large Transformer models themselves are typically "generalists" already, so structured generation and constrained decoding have given you the ability to use an LLM as a general classifier for years. It's an incredibly common pattern for working with LLM judges or any sort of branched decision making workflow.
A lot of people who are a bit less familiar with the field saw the hype around Jev and presumed that the reason it was so exciting was that it was a fundamentally new interface for working with an LLM. And that additional excitement drove even more attention to Jev. But fundamentally, TypeSafe's announcement was that they found a particular architecture/training paradigm that resulted in a model for this particular interface that had incredible accuracy, very low latency, and for which they could offer inference at a super low cost.
I've not kept up with the flood of Jev clones that have been released, but I think this is just typical for any new component in deep learning that gets popular. There are an absurd number of open source autoregressive LLMs and fine tunes you can use. The thing that makes one more popular than the other is typically the general performance of the individual model.
But training a model for this purpose, or emulating the procedures described in Jev's papers, isn't something that would be beyond the capabilities of any lab. It's not an entirely alien architecture or approach.
The bigger question for TypeSafe as a company would be if other teams are producing Jev-like models that win on performance or cost. Like I said, I haven't followed the reports super closely, so no idea if that's the case or not.
If CF's benchmark is representative and sufficient, Clef outperforms Jev!
Models by themselves don't guarantee market capture. Rather, its how they integrate. I think a lot of folks are burned by the closed nature of many models.
Now that we're hitting against the hardware supply limits of global economy, I expect more people to go back and revisit the things left along the way in the mad rush to "just throw more compute at it / make a bigger model" - and thus many more cases like Jev to show up in the next few years.
The hard part is the data and evaluation. Sure, it’s not that hard to build a fast model with good predictive power. But fast at doing what? You probably don’t care about classifying whether a hotdog is a sandwich (which is the Jev demo).
There is no official qwen 3.8 9b
16ms latency. And locally run.
https://at0m.pienomial.com/
Why go big when you can go small ?
Instead, to make up for the lack of economic viability of their models, they are forced to release publicly to get marketing to get others to pay based on hype.
Perhaps that may be too costly atm
And it's only been a few weeks.
There are many, many of those left around, because AI frontier is moving forward so fast, everyone is racing ahead. Which is why I laugh when people say AI is not transformative and LLMs are a dead end (and my favorite, "what are we going to do with all those GPUs when the bubble pops?"). Even if SOTA LLMs hit a hard capability limit tomorrow and never advanced again, there's a good decade of growth and advancement to be extracted just from all the low-hanging fruits that were left unpicked along the way.
Whether or not LLMs can self-improve their frontier capabilities, they can absolutely create a wake for themselves that accelerates everything else that's training on their synthetic data. We'll see every architecture of the past 40 years suddenly show leaps and bounds.
What you can do with LLMs is reverse this: from any numbers of snapshots of flight data, you can create large numbers of plausible user queries, based on your data, that are known to be feasible or infeasible. And now you have a labeled data set to train a model that focuses solely on the query-creation and judgment systems. And you can experiment with whether having a more flexible query protocol leads to higher success rates without sacrificing accuracy, or whether you can generate that last-mile feasibility check as a combination of auditable code checks alongside AI-based judgment.
LLMs don't absolve you of having to break down your system architectures into components that have well-defined boundaries (though certainly they can help with that design). They do make those components feasible to solve at scale.
then vision & robotics.
while everyone's chasing the frontier.
Good to see interest broadening beyond "just extend thinking." More approaches in the toolbox means fewer problems get treated as nails.
Your comment here made me laugh, because I think we will finally be able to play Crysis at 10fps.
Just kidding, of course. GPU half lives are quite a bit less than standard compute half lives, no?[0] That's what I've been trying to understand regarding data centers focusing as GPU clusters -- seems like the ROI window would have to be very short for the capitalization.
[0] https://www.tomshardware.com/pc-components/gpus/datacenter-g...
"Discriminative" always had pointlessly bad optics, but I knew it was over when I started seeing prominent machine learning researchers who p=100% knew better describe discriminative models as generative because that was the buzzword of the year. "Decision model" sells the value proposition much better and doesn't sound like an anti-woke crusade.
- I'd like a option, are you sure, not, repeat
- in and out of doors
- sisyphean effort in the cave
- jev-ish level grinding
It was impressive, beat pomemon for less than $2, but not all that interesting. People asking how different Math.Random plays pokemon would be
"Clef builds upon this concept, but uses a different base model as the backbone. We currently use Qwen as the base model and post-trained it to suit decision model use cases. "
"In the same week that Jev came out, we posted about some experiments [1] we had with our own homegrown decision model."
[1] https://x.com/michellechen/status/2101091012559151480
As far as I've seen, all of the Jev-compatible projects simply take an LLM, hack off a layer or two at the end, and call it good. Some of them spend more work than others trying to back-estimate in accurate probabilities.
A) Structured outputs from LLMs (doesn't need fine tuning but can be expensive)
B) Classification output from fine-tuned BERT-like or GLiNER models (is calibrated well and has cheap/fast inference)
What Jev did is combine the advantages of both A and B into one model/product, and create a really good API.
They claim that a key innovation is how they've trained the model using what they call RLCD (RL from calibrated decisions). So it's not just that you can get the outputs (which is easy to add to any LLM) but that the different primitives they expose (Choice, Score, Noul) have each been calibrated. For example, they claim that if you use the Score primitive (which gives you probabilities along a bunch of choices representing a continuum) that's not just using the more general 'Choice' primitive under the hood. It's been calibrated separately.
I don't know how many of the Jev-like things we've seen do that. For example Cloudflare offers a Jev-like model with the same API, and which they say was trained with RLCD. But I don't know whether Choice and Score are different under the hood, or whether Score is just sugar on top of Choice. (Should be easy to test this, but I haven't done it.)
You make it sound like this is some noteworthy timeframe. I'd be more surprised if Jev wasn't immediately made obsolete within days of release (which was effectively the case), especially when backed by a company like Cloudflare lol
ML moves quick. Add the extreme hype and cash floating around in this space and you can expect that anything resembling something novel and relatively untapped is going to be pounced on and turned over basically immediately.
With Jev, structured output is its native format. Jev is just way faster, cheaper, and outright better for a lot of things.
Note that while all of this is great, this is nowhere near a sort of "ChatGPT moment". It's a cool new thing, and it's way better at certain tasks, which could be big. That's all though.
The weights have permissive licensing, but the data and training pipeline are not published to reproduce them from their proprietary Qwen starting points. Weights are not "source."
I built this for my own needs, and thought others might find it useful too.
I'd love an privacy first on-device model i could use in iOS.
At0M: A 60M local Jev at 16 ms latency and 79% accuracy on Typed Decision
https://at0m.pienomial.com/ https://news.ycombinator.com/item?id=49920350
At0M: A 60M local Jev at 16 ms latency and 79% accuracy on Typed Decision
https://at0m.pienomial.com/ https://news.ycombinator.com/item?id=49920350