HN Explorer

Ask HN: How do you track NLP models in prod?

1 points by Nevin1901 ↗ HN
Working on a service which uses GPT-3.5 and was wondering how I would go about tracking the usefulness of it. Wondering what you guys have done, or if you guys even track your NLP models at all.

5 comments

[ 2.6 ms ] story [ 18.3 ms ] thread
PaulHoule
Are you doing classification, regression, extraction or something different? How did you evaluate the model before it went into production?
Nevin1901
Causal text generation. I evaluated it on my own internal dataset but real world data shifts over time.
PaulHoule
How do you get a number for text generation?
Nevin1901
Exactly my question. I just scanned over the inputs and outputs and compared how "useful" they were.
PaulHoule
Treat it like somebody grading an essay. It is either pass/fail or you can grade from 0 to 100 though you probably want to develop some kind of scoring rules for the latter.

I know a lot about evaluating classification but I am seeing people struggle w( evaluating text genersators; why don’t you look up my profile, send me an email, and we can talk more.