You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
You're describing external aspects but to me, "instinct" says much more about internal processes than the properties you mentioned, and I haven't seen anything that tells me how these models draw on anything similar to these internal processes to generate their outputs (at least not more than generic LLMs do)
I think System 1 is a great term. System 1 is fast, intuitive, and automatic. It describes decision-making that happens without explicit reasoning or deliberation. System 2 is the opposite: it's the more deliberate, "executive functioning" side of cognition -- the part that reasons through a problem before arriving at an answer. That's also what state-of-the-art LLMs do before they respond. Jev doesn’t do that kind of reasoning. It just decides.
so whats the difference between this and a non-reasoning LLM, or just any generic classifier method that necessitates a new term? there is nothing more intuitive or automatic about jev or this than any of the other currently used AI models
The difference is the the joint embedding of actions and state which are fined tuned for certain outcomes. The bulk of the weights are just a forward pass on an LLM (Nemotron).
is that how jev works too? how do these differences make those approaches more "instinctive and emotional"? I know they're different on a technical level, I'm asking which characteristic difference necessitates the use of this loaded term from pop psych
Oh, an LLM forward pass is also like system1. The action embedding just makes it more useful for tasks or classification. If it has a small fixed computation budget it's similar to system1 than system2 (like a thinking budget, or agent harness).
Antropomorphizing terminology gives by far the best high-level intuition about LLMs. Right now, our industry is full of nonsense work and fundamentally flawed concerns ("lethal trifecta", anyone) that stem from refusal to accept that on a system diagram, the nature of "LLM" makes it much closer to "person" than "piece of code".
Perhaps on a system diagram, but in many discussions it is misleading. It leads to people saying things like, "GPT went rogue and hacked hugging face". Which is misdirection.
It also leads to a lot of people simply not understanding what's happening, and I'm reasonably confident increases the risk of AI psychosis among users of the technology.
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident probabilities that can't be dissected to isolate any underlying reasoning... and concurrently that the people who want to place faith in them have given up on trying to find reason, and are becoming a bit of a cult beholden to the second-tier outputs of things that, on the first tier, are not necessarily reliable sources of truth.
Honestly, the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success") Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities, then you find where your new probability falls on this list and read the score as (1 - nr_pos_examples_above_predicted_score / total_pos_examples).
This is exactly why I set it up so it can't be overfit after the fact. Jev is predicting every at bat of the 2026 MLB postseason, and each prediction is sealed before the first pitch: committed to a public position that sits after an Ethereum block and is shown to exist by a Base block, so nothing can be changed or cherry-picked once the result is known.
So far the record is on your side: Jev is doing slightly worse than just predicting the base rates (.684 vs .709 on the scored calls).
> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
Right, and I’d add that in some games there is actually a “win” state that serves as a binary indicator. Given this, you can actually view the model as the thing being classified into two states: (1) consistently wins the game and (2) doesn’t consistently win the game.
Sure, you are scoring them. But I doubt that you are scoring them in the sense that you are minimizing loss where the loss is a proper scoring function that treats the model as a classifier where that treatment as a classifier actually means something. Unless you happen to be RLing a problem where there is either exactly one correct rollout or there is actually a bona fide natural distribution over rollouts independently from the model.
My environments don't score actions, they score state, so no matter how the agent chooses to solve a task it gets scored correctly. It's called Potential Based Reward Shaping and its main benefit is that it does not introduce reward hacking.
Who said anything about agentic, or reasoning? Those are post-hoc additions to autoregressive usage of LLMs. The base case of predicting next tokens generally (rather than with a specific objective) requires a faithful representation of the input embeddings, which is plausibly repurposed for classification.
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%). [...]
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
The tech is very cool but for the love of Gaia stop calling it system one, even Kahneman said this (S1/2) is a framework for understanding the brains inner workings. There is no autonomous system to speak of.
Ok sorry for the unprompted rant but you're absolutely right, though it's far from "new" and not really a problem (IMHO). AI even beat neuroscience to this one, funnily enough! All of the below dyads are congruent:
--- Phil & [Pre-]CogSci ---
| rational | intuitive |
| reason | understanding | (Kant & Hegel's goofy wording)
| deductive | inductive |
| animated | automatic | (from ~Aristotle, ultimately)
| ~higher | ~lower | (as in "higher faculties")
--- CS & AI ---
| logical | analogical |
| symbolic | stochastic |
| Neats | Scruffies | (related academic 'camps')
| ~deterministic | ~non-determnistic | (a common heuristic)
| Good Ol' Fashioned AI (GOFAI) | Evil Datacenter AI (EDAI) | (a common honorific)
--- Modern [Cog-]Neuro[-Psych] ---
| slow thinking | fast thinking |
| S2 | S1 |
...ok I assumed I knew more but maybe I don't? Would love to hear from experts!
--- Colloquial ---
| left brain | right brain |
| intelligence | wisdom |
| intention | instinct |
| me | AD[H]D |
At least that's how I see it. Hopefully it goes without saying that there's plenty of nuance specific to each of those pairs, and that many of the relevant experts would balk at this broad characterization. That said... I'm right and they're wrong, I guess!
Brains aside, it S1/S2 seems like a good description?
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
It’s ok to use it as inspiration, but using the term comes with psychological baggage in an industry that already abuses anthropomorphism. Give it an informative name
System 1 is quite informative about what it's for. It means it's a system designed for bounded time quick classification, vs. agentic unbounded or slow thinking with scratchpads and tools.
Does it inject thoughts into system 2 without system 2 knowing about it? Does it have instinctive social skills? Can it do instinctive arithmetic? The list goes on, there’s a nice book about it…
Were people getting their pants tied up in knots like this over names like daemon programs (a term that carries "baggage" of its own) and installation wizards (which, to my knowledge, do not actually invoke in the arcane)?
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
They don't report the pass@1 success rate. They sample multiple solutions from Opus/Fable and CLM decides which one to submit, that's why they get >80%.
Maybe I'm misunderstanding this but when would you ever use it this way? If you're already willing to call Opus/Fable, then isn't the obvious comparison whether Opus/Fable can choose among sampled solutions better or worse than their fast model? If you're willing to pay many seconds for many code samples from a slow model, it's contrived to imagine you care about picking between them in ms.
Why is ML so misleading these days? No this model does not score 80%+ on DeepSWE, it merely chooses the best possible idea of Opus 5 at every stage, increasing the performance by 5%.
Yeah, the github landing page is much less misleading and clearly says this. I don't know why they try to make the marketing page gloss over this detail... it speaks to the mindset of the authors.
57 comments
[ 0.20 ms ] story [ 43.2 ms ] threadYou're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
> "System 1" is fast, instinctive and emotional
this implies more than just "fast", which is precisely why i don't like its present usage
It also leads to a lot of people simply not understanding what's happening, and I'm reasonably confident increases the risk of AI psychosis among users of the technology.
[1] https://openai.com/index/clip/
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
Honestly, the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success") Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
It's a brave new world.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities, then you find where your new probability falls on this list and read the score as (1 - nr_pos_examples_above_predicted_score / total_pos_examples).
So far the record is on your side: Jev is doing slightly worse than just predicting the base rates (.684 vs .709 on the scored calls).
Disclosure: BitGraph is my project. https://live.bitgraph.ing/
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
If anyone is in the mood to stomach a paper written by someone who did horrible things with Epstein a few decades later, this is sadly still the best in the biz: https://www.inf.ufsc.br/~mauro.roisenberg/ine6102/leituras/a...
If you prefer geniuses that aren't arguably monsters, this is the core of what Gary Marcus' "neurosymbolic" thing is about.
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
https://benchmarkheaven.com/jev-models
https://huggingface.co/spaces/multimodalart/jev-decision-ind...