Show HN: Waffie, an LLM API automated testing tool (github.com)
Hey yall,
Very early release but wanted to get the idea out there.
Waffie is built for developers looking to operationalize LLM APIs into their solutions.
The first thing I'm tackling is automated testing of LLMs in the context of unstructured to structured data interpretation. For example, sentiment analysis.
Specify what you want in a YAML file and a test.csv and run waffie against your test cases.
Looking to bridge the gaps between software development and machine learning engineering.
4 comments
[ 3.0 ms ] story [ 20.8 ms ] threadAdd to the top of the readme something that basically shows "put in this / do this" and "get this"
Kind of like a before and after.
Basically, I'd recommend exposing 2-3 of the examples on the readme, otherwise the readme just shows how to run it, which doesn't show what it does / what it's for.
So you'd provide it for example with a csv with prompt and expected completion, and then a test file, and then you can run that test file and see how it does on those sample prompts.