HN Explorer

10 comments

[ 4.4 ms ] story [ 28.1 ms ] thread
skybrian
Okay, but how would I write evals for my project's agents file? Any good examples out there?
stuaxo
I mean.. Claude kept putting in deprecated APIs for code I was getting it to write, so I adjusted the prompt to say not to + it seemed to help.
sjmaplesec
Can add this as a skill or as part of a skill, and so you don't need to keep prompting the same things.
hamuraijack
so how would you eval your own claude.md? Each context is unique to the project, team, and personal root claude.md. Do you just take given task and ask it to redo the same one over and over again against a known solution? Do you just keep using it and "feel" whether or not it's working? How is that different from what everyone is already doing?
sjmaplesec
The review eval tests language, activation etc of skills. I guess you could move it all to a skill quick and then run an eval on that if using Tessl. This checks if the way you write the instructions etc are being well understood by the agent
furyofantares
If it was easy to write evals, I would come at it from that direction.

But since it's not, what I do to avoid working on AGENTS.md blind is I test it on whatever causes me to write it.

I have some prompt, the AI messes it up in some way that I think it shouldn't, maybe it's something I've seen it do before and I'm sick of it. So I update AGENTS.md, revert the changes, /undo in the chat context and re-submit the same prompt.

sjmaplesec
Tessl can generate the evals, both to test anthropic best practices as well as running scenarios with and without the skill to check if it's helping