I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.
Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.
JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.
A full run asks 534 English decisions.
The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.
Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.
Interesting; was curious how this didn't fall into trouble with ToS. Apparently the "no benchmarks" clause was intended for "limited preview" audiences and didn't get removed at launch on accident.
Many SaaS vendors forbid benchmarking, I find it crazy that such anti-competitive terms are standard across the industry but they are. Generally the goal of such terms is to "control the narrative" around the product, regardless of the truth of performance being better or worse than competitors.
Keysmashing my keyboard resulted in 86% confidence that the text was AI written. I don't think this is a particularly good classifier, I've never seen an LLM output "kad jfkhasljkdhf laksjhdf".
Edit: If that's not realistic enough for you, the text "Hello world! My name is GravitasIsOverrated and I like coding and cooking. This text is 100% genuine, and not AI generated at all." results in 85% confidence that it's AI generated.
I asked free chatgpt to give me some essays that will fool a slop detector and they all fooled this slop detector. Its best guess was "6% slop probability 87% confidence answered in 0.5 s for $0.000027" and yet it was 100% slop.
I am very skeptical slop detectors will ever work.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
Isn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so we don't know... "Trust me bro Jev is revolutionary and amazing, pay more money for our inferior product which costs more to run, and of which you need to access by sending us the data"
You can spot vibecoded websites by how they include the prompt or commit-style comments into the literal interface, instead of communicating it via visual context (or simply excluding it)
> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.
A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
It's fine in isolation to have help text for complex interfaces needing explanation. Better yet contextual hints.
But there's about 10 other examples explaining intention of the coding rather than immediately useful information to the user, it's all over this one site in small grey text. And I guarantee you nobody is reading them carefully. Just like how nobody likes reading a 15 line LLM code comment over a simple function.
Most of it could be better solved with more thoughtful design or deleted. The link explanation is particularly egregious.
If you don't have any kind of agent instructions saying "don't copy code into prose", you'll inevitably get text that reflects some previous state of the system instead of its current state.
One person’s helpful metadata is another person’s noise. It’s much better for a door to visually indicate that it should be pushed open than to put up a sign there.
However, doing the former requires a level of empathy with humans that LLMs rarely have.
Human brains have caloric demands. It is possible for humans to process enormous amounts of unrelated facts to make a decision, but it’s tiring. It’s much better to not do that, especially just to get some basic information.
To anthropomorphize a bit, an LLM might find it charming and interesting to read someone’s life story as a preamble before their taco recipe. Humans by and large find that annoying, not because we can’t understand the biography but because processing that information is not free.
So it’s probably possible to design using an LLM. You would probably have to be intentional about it.
"instructions": "Which intent does the user's message express?",
"labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
"state": "Play some Taylor Swift.",
"expected": "play_music"
20 comments
[ 2.6 ms ] story [ 27.9 ms ] threadI built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.
Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.
JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.
A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.
Leaderboard right now:
#1 - Jev 74.4 #2 - SemIf 73.1 #3 - djev 73.0 #4 - Winnow-12B Q8 71.2 #5 reflex 4B 70.3.
MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes: https://github.com/fstandhartinger/jevbench
Two no-signup demos: https://who-is-right.app.mintapis.com https://is-it-ai-slop.app.mintapis.com
Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.
Wdyt?
"Trust, but verify" isn't just a catchy cliche. It's the only way to operate where models and code are fast to market.
We've been experimenting with Jev for classifying email, some thoughts here: https://housecat.com/blog/classifying-email
Flagging AI written email is a much requested feature too.
Edit: If that's not realistic enough for you, the text "Hello world! My name is GravitasIsOverrated and I like coding and cooking. This text is 100% genuine, and not AI generated at all." results in 85% confidence that it's AI generated.
I am very skeptical slop detectors will ever work.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.
A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
But there's about 10 other examples explaining intention of the coding rather than immediately useful information to the user, it's all over this one site in small grey text. And I guarantee you nobody is reading them carefully. Just like how nobody likes reading a 15 line LLM code comment over a simple function.
Most of it could be better solved with more thoughtful design or deleted. The link explanation is particularly egregious.
However, doing the former requires a level of empathy with humans that LLMs rarely have.
Human brains have caloric demands. It is possible for humans to process enormous amounts of unrelated facts to make a decision, but it’s tiring. It’s much better to not do that, especially just to get some basic information.
To anthropomorphize a bit, an LLM might find it charming and interesting to read someone’s life story as a preamble before their taco recipe. Humans by and large find that annoying, not because we can’t understand the biography but because processing that information is not free.
So it’s probably possible to design using an LLM. You would probably have to be intentional about it.
e.g.,