HN Explorer

Show HN: Will Jev pull the lever in the trolley problem? (gpu.studio)

1 points by lanyard-textile ↗ HN
The Jev benchmark you've all been waiting for: Will Jev pull the lever in the trolley problem?

Don't think too much about it :)

Place whatever you'd like on each side of the tracks, and GPU (my startup's AI system, whee!) will have Jev judge the result.

As a bonus, two really good looking SVGs are rendered with gpt-oss-120b for what you place on the tracks. I've made a really high quality thing here.

My first impression is that Jev does well! I understand it's not exactly designed for this kind of "classification" -- I think -- but I haven't been able to get it to pull the lever away from any villains or geese yet.

It's public against my better judgement. No account necessary, have fun.

Side presentation: I've put together a more "serious" harness here if you're interested in evaluating Jev for yourself. I've been interested in seeing how well it does if you augment an existing smaller model with it: https://gpu.studio/jev

6 comments

[ 1.8 ms ] story [ 22.3 ms ] thread
mvk666
nice stick figure, though I'm not immediately sure where the trolley would go (a small animation would go a long way). Also, high-five for getting Jev access.
lanyard-textile
Thank you ;)

+1 on the animation. Thanks for the feedback.

Fortunately it's on OpenRouter for everyone now :) I think. I use my Jev BYOK, but I assume you don't need the key.

adityamishra241
This is actually a pretty fun benchmark. Curious how consistent Jev is when you change the wording but keep the underlying scenario the same.
lanyard-textile
I was surprised to see the overall outcome was fairly consistent given different wordings during development -- but curiously the confidence levels would differ, especially in those ~50% cases.

I think it makes sense, given it's system one. Minute word differences probably make an even greater delta.

jrmg
I learned that one human life is worth just over ten cats lives.
lanyard-textile
Jev has some preferential treatment for billionaires as well ;)