I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764
From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."
I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.
There's no need to, the 50 foot view is simply that many alternatives exist and they mostly fall into 3 meaningful weight classes with comparable performance among each class's members: too expensive to use indiscriminately, too big to run at home, and too small for complex work. As for names and faces in between, the overarching conclusion is that we're rapidly approaching commodity status and those don't really matter much
I'm working on solidifying local LLM work flow so I can stop tweaking the nobs and build things that arn't AI related.
There's tons of people caught up in the AI gold rush, but the value of real products for real people is out there. Even more so when you exclude the "I slapped AI onto this"
Can’t say I’m a fan of containers for this. A big chunk of local LLM gains come (imo) from the open modular nature of llama.cpp and friends. Easy to modify. Easy to experiment.
Containers are the proprietary binary blob in hardware world equivalent
What? How are those even related with each other? You can just as easy modify and experiment with llama.cpp in a container as outside of it, they really shouldn't impact one another. Containers don't suddenly make llama.cpp less "open modular" somehow, and I'm not sure how you'd arrive as such conclusion.
There is not a single person mentioned on the website, github created 3 days ago, no real contact, everything hidden. Completely anonymous. Domain owner hidden.
hey guys, founder here. I am working completely solo right now running on grant funding, so things slipped through on the website quality, it is largely written by AI, I'll deslopify the blogs and be much clearer about methodology. Neutrino-1 is a starting point for Fermion, It's a density experiment on how much capability survives per byte. rn working on really fast and efficient TTS and realtime models to which this architecture is generalizing pretty well.
Sorry for all the confusion from reading through the blogs, should be fixed in a couple of days!
23 comments
[ 2.8 ms ] story [ 38.1 ms ] threadPrismML actually targeted the same Qwen 8b model and got it down to 1.75gb here: https://prismml.com/news/ternary-bonsai
I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764
From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."
The section from huggingface is here: https://huggingface.co/blog/1_58_llm_extreme_quantization#fi...
I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.
I get that it’s designed to run on a CPU, big GPU or MacBook (although the way that was phrased confused me at first).
I’m struggling with what a “decoder-only” model is good for.
And if it’s decent today, it’s shit in eight months! I tool hop as much as the next dev but this is a bit much.
There's tons of people caught up in the AI gold rush, but the value of real products for real people is out there. Even more so when you exclude the "I slapped AI onto this"
Containers are the proprietary binary blob in hardware world equivalent
Blog populated with incoherent PR material generated by Yet Another AI.
Sigh...
Only problem was there is plenty of space on the device. PLENTY (not quite 750gb).
Sorry for all the confusion from reading through the blogs, should be fixed in a couple of days!