Fine-tuned small LLMs can beat large ones with programmatic data curation (tensorzero.com)

I’ve been thinking about curating primary sources themselves and then using those for fine-tuning.

Anyone gone that route and know of projects with very high quality curated source materials? ideally categorized and labeled.

Maybe this is a nitpick but CoNLL NER is not a "challenging task". Even pre-LLM systems were getting >90 F1 on that as far back as 2016.

Also, just in case people want to lit review further on this topic: they call their method "programmatic data curation" but I believe this approach is also called model distillation and/or student-teacher training.

Is this just distillation but with a step to filter out low-quality responses first?

Noob question: Would it be possible to train a small model for a single prompt?

I think its a good idea but how do you not accidentally benchmark hack here?