Why outcome-billing makes sense for AI Agents (valmi.io)

So who's the arbiter to determine if the outcome was achieved?

And how do you programmatically measure it?

Outcome billing is ideal for pretty much any SaaS product.

Sounds great in theory, until you realize everyone has a different definition of outcome.

It really makes sense, and the best part — customers love it. It’s the simple form of pricing, and it’s simple to understand.

In many cases though, you don’t know whether the outcome is correct or not but we just have evals for that.

Our product is a SOTA recall-first web search for complex queries. For example, let’s say your agent needs to find all instances of product launches in the past week.

“Classic” web search would return top results while ours return a full dataset where each row is a unique product (with citations to web pages)

We charge a flat fee per record. So, if we found 100 records, you pay us for 100. Of its 0 then it’s free.

Maybe it's not as nice a story there as he's from India, but outside India people like to talk about their cobra problem and failed solution (retold below). This feels like that. If it's a ticket system, it could close them all as unresovable overnight. If it cares about customer satisfaction, it could give everybody thousand dollar gift cards. Point is, AIs existence is predicated on finding a way to improve its score by any means necessary, and that needs very careful bounding.

I believe it was under British rule, they offered a reward for people bringing in dead cobras as proof of culling. Which worked until people started breeding them just to get the reward. Humans gamed the system and it made the problem worse.

You can apply the same philosophy to employees and if you dare to do so you will quickly find out that it does not work. When a measure becomes a target, it ceases to be a good measure - Goodhart's law. I cannot see why AI agents should be treated differently when it comes to fuzzy measurements of performance.

Is this actually different from just guaranteeing some metrics? Like if you have a document processing “agent” that extracts fields from forms, you’d have an accuracy threshold and have some checks set up to verify this?

Does “outcome billing” amount to anything different?

I started reading the article and immediately got hit by the incorrect statement in the opening:

> If AI agents help each support employee handle 30% more tickets, that's like adding 30 new hires to a 100-person team, without the cost.

I think this is an oversimplification designed to make LLMs seem more profitable than they actually are.

The one wrinkle this might have is that it incentivizes the agent developer to over-resolve or “over outcome” to ensure they hit targets.

This is risking the end customer experience for your Agent buyer, which might not be worth the risk to a company that wants to keep customers very happy.

Outcome billing may seem to make sense for AI.

Maybe the pricing model makes sense in the beginning.

Until people will realize the big secret - AI is still just software.

A new category of software.

The price of software generally only goes in one direction, and that’s a race to the bottom.

Outcome-billing makes absolute sense! In every case where I have used an LLM to work on a software project, I have been frustrated by the process and end up educating the thing myself. The outcome is that it has learned from me, so I need a place to send my consulting bill.