If the author would like, I self computed a j lens for the 27b version of the qwen model. I used it for my own exploration in this area, and can share it if you want.
This sounds like a great foundation for an adtech startup.
If you provide free chatbot services, but sell advertisers bids on which steering vectors to use to bias towards products, based on an embedding of the prompt, I bet you'd make a ton of money.
> Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
But ultimately this is just an engineering problem, no?
Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.
It might be easier, but I experimented a bit, and the prompted writing always felt a bit heavy handed; it tended to leak that mentioning the product was prompted. For ads, I think you want something a bit like Golden Gate Claude, if anyone remembers that experiment:
" the intermediate activations of an LLM to decode what it is most likely going to say or is thinking about."
There is no thinking in these models. The J space is a basic technique measuring how much the influence of shifting a token earlier changes it later. Anthropic can wrap it up in a 100-page paper peppered with language about 'consciousness' and other, but that is basically the gist of the entire method.
Discussing whether models "think" is impossibly confounded by conflicting definitions of that it means to "think". All this noise about "thinking" isn't really about what models can do, it's about what what every participant in the conversation privately thinks "thinking" means, but we're not all using the word the same way.
So when you admonish someone to say "there is no thinking in these models" while not clearly defining exactly what you mean by "thinking", your assertion that models don't do it are meaningless at best, and false or deceptive at worst.
15 comments
[ 0.23 ms ] story [ 4.9 ms ] threadIf you provide free chatbot services, but sell advertisers bids on which steering vectors to use to bias towards products, based on an embedding of the prompt, I bet you'd make a ton of money.
Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...
Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.
https://www.anthropic.com/news/golden-gate-claude
> If you ask this “Golden Gate Claude” how to spend $10, it will recommend using it to drive across the Golden Gate Bridge and pay the toll.
There is no thinking in these models. The J space is a basic technique measuring how much the influence of shifting a token earlier changes it later. Anthropic can wrap it up in a 100-page paper peppered with language about 'consciousness' and other, but that is basically the gist of the entire method.
So when you admonish someone to say "there is no thinking in these models" while not clearly defining exactly what you mean by "thinking", your assertion that models don't do it are meaningless at best, and false or deceptive at worst.
You can dismiss the criticism with pedantics about what he means by 'thinking', but what I said is very clear if you read the underlying paper.