Ask HN: When is fine-tuning a small LLM worth it?

It's worth it if a) you have a decent sample size of data for your problem and b) you have a cost-effective infra to host it.

Notably the latter is more of the bottleneck, particularly with the price race-to-zero with models such as GPT-6 Luna.

Maybe when you want to learn the process for when/if you ever need to fine-tune a larger model?

It's faster to make iterate when you're toying around with a 1B model than a 27B one.

I've trained two small models, one I attempted to train it to be an expert (decent?) hearts (card game) player, and the other I trained on dream interpretation texts, I wanted to see if a model could interpret dreams reasonably well if it were trained on all the literature in existence (that I could find anyway). The hearts experiment was a general failure because the game of hearts as played by humans just has too many strategies to consider. I trained it purely on games (~20,000 hands of hearts in the RL dataset I generated) and the trained model did about as good as any decently written procedural game logic would have. the dream analyst training was a bit more successful. I can hand the model a dream log and it will offer some interesting insights. are they accurate? hard to say, it's a very subjective type of training, but definitely a better experiment than the hearts expert training run. in both cases the biggest and most time intensive part is generating the training dataset. building the LoRa is the easy part.

I was using Qwen3.5:2b models for both, running on Dell Pro Max GB10 Cuda,128GB.

I personally would only do it if I need it to solve a highly specialized problem that requires every inch of privacy I can get. Otherwise there are better options out there

I fine-tuned a model for my first AI web app back in 2019. It created plausible reviews of artwork from prompts. They were totally hallucinated though and had nothing to do with actual images. It was interesting for language and bias analysis above all else. But, it was enough of a novelty that I got thousands of users. I have considered going through the whole process from scratch with Mistral to make a bespoke model and correlate it with a modern vision model. But, I don't want to spend any more money on this project. Hosting an unmonetized public facing AI app for 7 years was not cheap.

https://artreviewgenerator.com/

[dead]

[flagged]

I think it's almost never worth it now. An up to date LLM with a good few-shot prompt, or an agentic LLM will likely outperform a small finetuned LLM, at a lower cost with more flexibility.

Moreover, it's very seldom that you have enough high-quality data to do a meaningful fine-tune.

I have successfully finetuned a small Gemma to play chess badly, and that worked. Mostly because chess data is plentiful and because my aim was to not have perfect answers. I'm self hosting the model inference because running it in the cloud would be too costly.

I also finetuned small LLMs using too small datasets of business specific human conversations and the results were disappointing.

[dead]

I did a project as few years back - it was people categorizing construction defects from a form into a drop down. A human could do this fine, and while there were instructions, it was not clear until you saw how others did it.

AI would still not solve this problem properly today, and may in fact be worse the more data it's trained on.

Another problem it does poorly today is writing in the style of another. It might be the whole copyright adherence but it refuses to follow certain styles. If you give it examples and ask it to break it down into a prompt on how to write, it will still do a poor job. It will work if you actually put examples in the prompt, but at this point, you might as well fine-tune it.