It's the digital equivalent of Morty speaking with the death crystal: https://youtu.be/YjepJlvkdKs?t=51. The crystal shows him how he will die, so he iteratively determines his speech based on whether he sees himself dying with the life he wants.
We all have a death crystal, it's just wrong sometimes. It may confuse certain possibilities with death, when in reality they are a temporary discomfort or setback.
If you ignore it and you survive, it eventually recalculates and you stop seeing that future.
In the olden days when the text-davinci models would just wholesale make shit up they did some pretty funny completions.
An early ChatGPT model made me a pretty funny track list for my imaginary Indian cover band The Needful Dead. It's no longer in my chat history though so clearly Sam retconned anything his models did that could be considered racist.
Codex and I were trying to infer why Jev gave a certain parameter a given score. So I asked codex to produce a list of like 30 plausible reasons Jev might've selected that and then presented Jev with the initial prompt, followed by "You scored this with XYZ. What was your reasoning for doing this?" And then allowed it to do a noul value for each of the reasons codex generated. Felt like those people that give their dogs the buttons to push.
Jev has taught me the same lesson three times over now.
When it first came out, I thought "this weekend, I'll do a little open-source Jev based on single-token prediction and the token logit output", but of course when it came to it, there were at least 5 that had already been done between me thinking that and getting around to it.
So I wrote up[0] what other people had done, but wasn't happy with how weak the benchmarks were, but in the time between writing the first word and the last few, two excellent sets of benchmarks had been written, so I was able to incorporate those. I published the article, and one of the authors of one of the implementations commented that I'd beaten him to doing the write-up he'd wanted to.
This morning I thought "huh, you could have some fun giving Jev a single letter or token at a time, turning it into a chatbot", but as the time of looking two people had already done this (and taken the gag further than I would have), and ... this is isn't either of the ones I'd found. I bet if you scratch the surface there already at leat 5.
Time from idea to output has dropped off a fucking cliff.
> I'll do a little open-source Jev based on single-token prediction and the token logit output
How is this idea generally working out in comparison with Jev? I'm curious, from what I read so far it seems like Jev is still beating this kind of thing.
But it's curious because it's not entirely clear why, from an architecture point of view for all we know that's exactly what they're doing. So it must come down to the quality of those logits, ie., model size and training details.
It seems to me that what most of these single-token-prediction projects are missing is that Jev seems to be claiming they predict well-calibrated probabilities. This is an incredibly valuable thing that LLMs simply can't deliver unless they are trained specially for it.
> How is this idea generally working out in comparison with Jev?
Jev clearly has _some_ secret sauce compared to doing the dumbest thing that could possibly work with Qwen. It's not clear how durable that advantage is against OpenAI wiring up Luna-5.6 and doing a minimum amount of tweaking, but I presume we'll know in a week or two.
Everything old is new again, huh. I remember people doing this back in the early llama days, restricting grammars to yes and no tokens or 0 and 1 and then classifying questions. Usually it was rather ass in terms of performance cause no model is tuned to reply that way and it was WAY out of distribution, and yet then it got turned into the main way to run multiple choice benchmarks, and then everyone benchmaxxed it. Doesn't the normal MMLU/Pro also just do the same thing, restrict the output to one token, top-k=1, and it has to be one of the choice letters?
I think the real difference Jev makes is the fast parallel decode, it just seems rather bizzare how that works.
I think it would work well as a multi-emoji compiler too. Some phrases pair well with N emojis, and you’d be able to produce arbitrarily many emojis per phrase
I saw a lot of ppl think about what jev could use under the hood and could someone explain why this can't just be an embedding model where we just embed all the input + decisions and give back the cosine (or whatever) similarities?
if you compare an embedding model to something like Jev which asks 100 questions and use the answers as the embedding you will be able to get move mileage out of the latter.
that said, I don't understand the hype. I have been doing what Jev does for 2 years now by just forcing json tokens onto an LLM. you can even get the LLM to think. and you can ensemble multiple LLMs.
> I suppose the appeal of Jev is how cheap and fast it is
Ok, so you do understand the hype. I mean cost & speed are reaaaaally big issues for normal LLMs. If you can fill a specific use case way cheaper and way faster, that’s a fantastic development. We need more niche tooling that’s more efficient and better for niche use cases. Not everything has to be general purpose.
Not clear to me that playing games is the point of jev. This post is just a random fun experiment someone did because they wanted to.
But I can ask 100 causally masked questions against common prefix, and get 100 answers, all in a single PP pass using any existing "classical" attention transformer model? Like, I had the impression that is what everyone was doing for classification already?
Is the difference "we did RL to tune logit distribution"? Because I really do not see anything new there. What is the difference?
Embeddings just convert the tokens to a vector that represents the text in an abstract semantic space. JEV goes a step further and actually processes the instructions/meaning of those embeddings to produce output, just not the usual series-of-tokens output we expect from an LLM.
I've been trying to understand it, and best I can summarize it as, it's kinda like a smart classifier model, but one that can classify answers to yes/no questions.
36 comments
[ 0.24 ms ] story [ 52.1 ms ] threadIf you ignore it and you survive, it eventually recalculates and you stop seeing that future.
> a story
I think this is the first time I’ve knowingly laughed at a model’s joke.
An early ChatGPT model made me a pretty funny track list for my imaginary Indian cover band The Needful Dead. It's no longer in my chat history though so clearly Sam retconned anything his models did that could be considered racist.
When it first came out, I thought "this weekend, I'll do a little open-source Jev based on single-token prediction and the token logit output", but of course when it came to it, there were at least 5 that had already been done between me thinking that and getting around to it.
So I wrote up[0] what other people had done, but wasn't happy with how weak the benchmarks were, but in the time between writing the first word and the last few, two excellent sets of benchmarks had been written, so I was able to incorporate those. I published the article, and one of the authors of one of the implementations commented that I'd beaten him to doing the write-up he'd wanted to.
This morning I thought "huh, you could have some fun giving Jev a single letter or token at a time, turning it into a chatbot", but as the time of looking two people had already done this (and taken the gag further than I would have), and ... this is isn't either of the ones I'd found. I bet if you scratch the surface there already at leat 5.
Time from idea to output has dropped off a fucking cliff.
0: https://sgnt.ai/p/jev/
How is this idea generally working out in comparison with Jev? I'm curious, from what I read so far it seems like Jev is still beating this kind of thing.
But it's curious because it's not entirely clear why, from an architecture point of view for all we know that's exactly what they're doing. So it must come down to the quality of those logits, ie., model size and training details.
It seems to me that what most of these single-token-prediction projects are missing is that Jev seems to be claiming they predict well-calibrated probabilities. This is an incredibly valuable thing that LLMs simply can't deliver unless they are trained specially for it.
Jev clearly has _some_ secret sauce compared to doing the dumbest thing that could possibly work with Qwen. It's not clear how durable that advantage is against OpenAI wiring up Luna-5.6 and doing a minimum amount of tweaking, but I presume we'll know in a week or two.
I think the real difference Jev makes is the fast parallel decode, it just seems rather bizzare how that works.
Maybe you are conjuring them into existence.
- Someone with ADHD
that said, I don't understand the hype. I have been doing what Jev does for 2 years now by just forcing json tokens onto an LLM. you can even get the LLM to think. and you can ensemble multiple LLMs.
Ok, so you do understand the hype. I mean cost & speed are reaaaaally big issues for normal LLMs. If you can fill a specific use case way cheaper and way faster, that’s a fantastic development. We need more niche tooling that’s more efficient and better for niche use cases. Not everything has to be general purpose.
Not clear to me that playing games is the point of jev. This post is just a random fun experiment someone did because they wanted to.
I honestly don't.
No code published currently, but if somebody is interested I can clean that up next weekend and throw it on github.
This looks more like a dating app simulation than a chatbot