Ask HN: Agent evaluations, what is everything I should know?

3 points by akira_067 7mo ago ↗ HN

I'm currently building coding agents, and wondering what the standard is for creating and running evals for most people? I gather that the tasks and their definitions will be dramatically different across domains and instances, so I'm not hoping for a one size fits all. Just... what actually works for you in practice?

2 comments

[ 4.0 ms ] story [ 15.8 ms ] thread

adastra22 7mo ago ↗

The capabilities of the tool matter more. Claude Code, Codex, Cursor CLI all have different feature sets. This usually determines the choice more than base model capabilities.

akira_067 7mo ago ↗

You think so? What features are differentiated enough to warrant that?

Seems like they all have tools for reading files, editing, running shell commands.

Cursor has linting access, but this can be added to claude with hooks I think.