This is definitely a useful exercise worth going through for the educational value before eventually surrendering and just using the big models owned by "unprofitable companies."
> On paper, this looks like a success. In practice, the time spent crafting a prompt, waiting for the AI to run and fixing the small issue that came up immensely exceeds the 10 minutes it would have taken me to edit the file myself. I don’t think coding that way would lead me to a massive performance improvement for now.
The models used in this experiment - deepseek-r1:8b, mistral:7b, qwen3:8b - are tiny. It's honestly a miracle that they produce anything that looks like working code at all!
I'm not surprised that the conclusion was that writing without LLM assistance would be more productive in this case.
While I'm always happy to see more people using open models, I was hoping the "playing" would be a bit more about actually interacting with the models themselves, rather than just running them.
For anyone interested in playing around with the internals of LLMs without needing to worry about having the hardware to train locally, a couple of projects I've found really fun and educational:
- Implement speculative decoding for two different sized models that share a tokenizer [0]
- Enforce structured outputs through constrained decoding (a great way to dive deeper in to regex parsing as well).
- Create a novel sampler using entropy or other information about token probabilities
The real value of open LLMs, at least for me, has been that they aren't black boxes, you can open them up and take a look inside. For all the AI hype it's a bit of shame that so few people seem to really be messing around with the insides of LLMs.
Deepseek R1 8B isn't famous for anything (except maybe being confused for Deepseek R1) and isn't by Deepseek anymore than me finetuning Llama makes me the creator of Llama.
11 comments
[ 3.2 ms ] story [ 32.9 ms ] threadIf something can not be reproduced from sources which are all distributed under an OSI license it is not Open Source.
Non public sources of unknown license -> Closed source / Proprietary
No training code, no training sources -> Closed source / Proprietary
OSI public source code -> Open Source / Free Software
These terms are very well defined. https://opensource.org/osd
The models used in this experiment - deepseek-r1:8b, mistral:7b, qwen3:8b - are tiny. It's honestly a miracle that they produce anything that looks like working code at all!
I'm not surprised that the conclusion was that writing without LLM assistance would be more productive in this case.
That isn't an open source model, but a quantized version of GLM-4.5, an open-weight model. I'd say there's hope yet for small, powerful open models.
For anyone interested in playing around with the internals of LLMs without needing to worry about having the hardware to train locally, a couple of projects I've found really fun and educational:
- Implement speculative decoding for two different sized models that share a tokenizer [0]
- Enforce structured outputs through constrained decoding (a great way to dive deeper in to regex parsing as well).
- Create a novel sampler using entropy or other information about token probabilities
The real value of open LLMs, at least for me, has been that they aren't black boxes, you can open them up and take a look inside. For all the AI hype it's a bit of shame that so few people seem to really be messing around with the insides of LLMs.
0. https://arxiv.org/pdf/2211.17192