HN Explorer

Ask HN: Are agent benchmarks the new credit ratings?

1 points by shreyassudhanva ↗ HN
I did some talking to engineers at new-ish startups and agent companies are buying benchmarks to prove reliability to their customers. I'm theorizing that it looks a lot like pre-2008 credit ratings: the party being rated pays for the rating.

If you were to buy, or even approve an agent product, how much do you care about these benchmarks? What gets it over the line for you?

3 comments

[ 6.8 ms ] story [ 18.7 ms ] thread
blinkbat
Tests and community are far better indicators
uliseis
I don't really think benchmarks are very useful than for wide marketing... If I were a customer that wanted to get some AI agents in my workflows, I would prefer the vendor to provide a small PoC delivering "real" results than looking to (possibly) cooked results in benchmarks (as you say, that one who pays gets the rating)...