Ask HN: How do you catch regressions when you change your AI agent's prompt?

1 points by yakshithk_ 2mo ago ↗ HN

i've been building agents for a while and kept running into the same problem: change the sys prompt, swap models, or tweak for the agent calls a tool — and something subtly breaks. The only way I found out was manually running it or a user reporting it.

2 comments

[ 2.6 ms ] story [ 20.8 ms ] thread

Sasha_2040 2mo ago ↗

Any AI is is non-deterministic by nature, this is more about statistics, but not dedicated test cases. So, even if you swich to the model which is better in metric results you still cannot guarantee it will be better in your specific use case or do the same. What you can probably do is to automate your important test scenarious and run each every time you change model or system prompt

bisonbear 2mo ago ↗

I've been building a tool to do this - build a dataset based on tasks from your repo, then A/B test the agent with whatever change you're making to determine the impact prior to actually shipping it. If you want to check it out - stet.sh