Biggest gap I see in most "LLM for practitioners" guides is they skip the evaluation piece. Getting a prompt working on 5 examples is easy — knowing if it actually generalizes across your domain is the hard part. Especially for analysts who are used to statistical rigor, the vibes-based evaluation most LLM tutorials teach feels deeply unsatisfying.
I’m always curious why local models aren’t being pushed more for certain types of data the person is handling. Data leakage to a 3rd party LLM is top on my list of concerns.
Would that book be useful as a reference to introduce data journalism students to AI? I'm less interested in the basics of using the API or claude code etc than best practices for workflows dealing with unstructured data, entity extraction, automated pipelines (with evals)? Although I do have some decent workflows around this I'd be interested in reading from someone who lives and breathes this kind of work. Pure data analysis to me is also something where I haven't found a good bridge between the current "generate a python script for me that I'll double check" paradigm and the spreadsheet centric world of most data journalists.
8 comments
[ 89.1 ms ] story [ 338 ms ] threadDoes this guide cover systematic eval at all?
Or is this actually a law enforcement related example?