Look at this poster [1] (its low-res, full res is paid). Also see this video for context [2] - it talks about deepseek's innovation, but explains attention well. Above I was pointing to the moment where multiplication…
Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector…
I know you're joking, but if you really wanted to, you can if you have a network of these, since you can build NAND gates from perceptrons. If you have NAND gates, then you can build any other gates from these and then…
The "ALL CAPS" part of your comment got me thinking. I imagine most llms understand subtle meanings of upper case text use depending on context. But, as I understand it, ALL CAPS text will tokenize differently than…
And now we’ve built LLMs - the biggest mirror of them all. The Internet and smartphones gave us countless ways to look at ourselves and see how others see us. And then LLMs helped us gaze at the sum of all that and even…
At first glance, this reminds me of how branch prediction is utilized in CPUs to speedup execution. As I understand it, this development is like a form of soft branch prediction over language trajectories: a small model…
Skimming through the source it seems to run 'car' and 'person' objects through llava with the following prompt: - "person": "get gender and age of this person in 5 words or less", - "car": "get body type and color of…
That's a great scene. I mostly remember it for the exchange the two had after they finished modding the game and it worked [0]: Red: "Congratulations, son! You have seen the future!" Kelso: "Yeah, yeah, you're so right,…
Look at this poster [1] (its low-res, full res is paid). Also see this video for context [2] - it talks about deepseek's innovation, but explains attention well. Above I was pointing to the moment where multiplication…
Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector…
I know you're joking, but if you really wanted to, you can if you have a network of these, since you can build NAND gates from perceptrons. If you have NAND gates, then you can build any other gates from these and then…
The "ALL CAPS" part of your comment got me thinking. I imagine most llms understand subtle meanings of upper case text use depending on context. But, as I understand it, ALL CAPS text will tokenize differently than…
And now we’ve built LLMs - the biggest mirror of them all. The Internet and smartphones gave us countless ways to look at ourselves and see how others see us. And then LLMs helped us gaze at the sum of all that and even…
At first glance, this reminds me of how branch prediction is utilized in CPUs to speedup execution. As I understand it, this development is like a form of soft branch prediction over language trajectories: a small model…
Skimming through the source it seems to run 'car' and 'person' objects through llava with the following prompt: - "person": "get gender and age of this person in 5 words or less", - "car": "get body type and color of…
That's a great scene. I mostly remember it for the exchange the two had after they finished modding the game and it worked [0]: Red: "Congratulations, son! You have seen the future!" Kelso: "Yeah, yeah, you're so right,…