I've been fascinated by a small mention in the 1.58bit quantization article that mentioned 0.68 quantization , which I believe to mean 0,1 instead of 1.58's -1,0,1. When I read https://www.reddit.com/r/LocalLLaMA/comments/1bpa6ol/unoffic... great experiment of making their own unofficial 1.58b quantization, I began to wonder if I could squeeze a vector down to 1 bit. And.. I can! (with some caveats in the discussion)
It was when I realized that XNOR and population count could basically score 32 dimensions at a time.
While this isn't ANYTHING like an actual quantized LLM, I thought it was a really nice proof-of-concept, and could be very useful for smaller machines running RAG applications.
1 comment
[ 1.8 ms ] story [ 15.0 ms ] threadIt was when I realized that XNOR and population count could basically score 32 dimensions at a time.
While this isn't ANYTHING like an actual quantized LLM, I thought it was a really nice proof-of-concept, and could be very useful for smaller machines running RAG applications.
My Code: https://github.com/bigattichouse/bitvector_research
My Write Up: https://bigattichouse.medium.com/dreamcoat-cosine-similarity...
NOTE: I'm not saying 30X faster than GPUs, but CPU implementations could be 30X faster. Good enough for little machines like Arduino at least.