Hobbyist harness builder here. Thank you for the article.
What would be helpful would be a cheap API call that would give me the energy cost estimates:
> Given some prefix-cache uuid, what is the cost (in mW of GPU energy) of adding X tokens that will branch off from what's been cached at offset Y tokens.
Suppose at some turn n, I read a file and pull it into the context. Now at turn n+m, I discover that the file has mutated on disk. The most energy efficient option might be any of these:
1. Replace the content of the file in the cache
2. Append the new file to the end of the cache
3. Add a thing to the context saying the file has been updated.
4. Compact memory then do option #3
As a harness builder, it's pretty hard for me to pick the most energy efficient strategy - I assume every model has a different cost for prefill vs decode and I also expect that a provider (an inference cloud) might have slightly different implementations that would impact how expensive a prefix-cache miss is.
Ofcourse, I also need to target aggregators like openRouter because as a solo hobbyist harness builder, it's really hard for me to track every cloud's unique API. Then there's the fact that I'm optimizing for lowest billed cost, not lowest energy consumption.
I'm waiting for the industry to standardize on a good way to express these in an API so that I can start doing these tradeoffs at runtime in my harness.
It's strange that people who have LLMs write entire articles for them often claim this. It might be true that you prompted it with your ideas, but the result is still in its own style, and style (and it variety) is why we prefer to read human text.
This article mainly argues around harnesses but a breakthrough for me was understanding why harnesses. Without a harness and only a raw LLM, exploring and modifying codebases is not feasible. It’s just text in, text out, and has no way of handling the context of the codebase (if large enough). Plug an LLM into a harness, and all of the sudden it has an exosuit that lets it funnel its thinking into discrete actions. It can now use strategies to manage that large context in pieces.
This is too reductive of a statement to have any meaning. You have re-defined a class of program to encompass all programs. The term “harness” exists to differentiate it from programs that are not harnesses.
depending on how we define harness, it's at its core a loop around llm calls with the tool field populated and a function per to use them after a round trip, easily fits into a single file in a crude setup
I know this sounds ridiculous, but try fine tuning a small local shell -> plain language model, then using the output of that for whatever you need on your end. You'll get the gist of what the frontier model is trying to accomplish, and can handle it with whatever harness workflow you want.
I wrote an article explaining what an agent and a harness are with some detail. There is so much untapped potential in harnesses still, I think because we do not take the time to parse exactly what they are.
I don't know - all of these cache hit rate advocacy is kind of goodhearting. Cache hit rates are less important than the accuracy of LLM answers and people seem to forget about it. For example I've seen articles how sub-agents poison the cache. Sure you can fill up your context with irrelevant data to have high cache hit rate - but that degrades the LLM work - sub-agents are about providing the LLM with a clean context - just the information needed to do its job, no distraction.
All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.
I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.
The thing that surprises me is that any of the popular harnesses break cache this way. Is it just an artifact of too much LLM code and not enough tests?
It's easy to break the cache and there's a number of very valid usecases (mostly context management such as compaction or tool shaking).
Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.
Getting 99% cache hit rates while providing a good user experience is the hard part!
the fun ones are putting the current date in the system prompt, iirc OpenCode has fixed this, such that if your session spans day change, your cache does not bust
In all fairness, only Xiaomi and Deepseek have a long enough cache time for this to matter. As far as I know, most providers use a 5m cache timeout so unless you're working exactly around midnight your cache is more likely to be nuked by a distraction.
One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
I recall this company now for their unintuitive/abstracted pricing model I have no interest in, but it does make clear why they would be helping customers with cache hit rates, that's going to lower bills on a per-request basis
I've never seen a case where mucking with the prefix to a long context pays off. Append a message to the session, do a compaction/truncation in the current session, or launch a sub-agent, all ways to inject new information from the harness without a big uncached read. I don't think that cache hit rate is very a useful metric to optimize for, much better to come up with an approximate cost model and benchmark task completions.
In my opinion, Harness has nothing to do with energy (ie cost). They all use the same conversational requests so the efficiency is mostly in the prompt (instruction and data) as well as the chosen model.
Choose wisely.
For efficiency, providers should cache the initial prefix (ie system prompt and tools) of harnesses, not sure if they do all that.
28 comments
[ 0.32 ms ] story [ 5.0 ms ] threadWhat would be helpful would be a cheap API call that would give me the energy cost estimates:
> Given some prefix-cache uuid, what is the cost (in mW of GPU energy) of adding X tokens that will branch off from what's been cached at offset Y tokens.
Suppose at some turn n, I read a file and pull it into the context. Now at turn n+m, I discover that the file has mutated on disk. The most energy efficient option might be any of these:
1. Replace the content of the file in the cache
2. Append the new file to the end of the cache
3. Add a thing to the context saying the file has been updated.
4. Compact memory then do option #3
As a harness builder, it's pretty hard for me to pick the most energy efficient strategy - I assume every model has a different cost for prefill vs decode and I also expect that a provider (an inference cloud) might have slightly different implementations that would impact how expensive a prefix-cache miss is.
Ofcourse, I also need to target aggregators like openRouter because as a solo hobbyist harness builder, it's really hard for me to track every cloud's unique API. Then there's the fact that I'm optimizing for lowest billed cost, not lowest energy consumption.
I'm waiting for the industry to standardize on a good way to express these in an API so that I can start doing these tradeoffs at runtime in my harness.
It's strange that people who have LLMs write entire articles for them often claim this. It might be true that you prompted it with your ideas, but the result is still in its own style, and style (and it variety) is why we prefer to read human text.
One level deeper is all numbers though.
And eventually it's all 0 and 1s!
/s
https://aweb.ai/blog/what-an-ai-agent-is-a-primer/
All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.
Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.
Getting 99% cache hit rates while providing a good user experience is the hard part!
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
https://neuralwatt.com/about
I recall this company now for their unintuitive/abstracted pricing model I have no interest in, but it does make clear why they would be helping customers with cache hit rates, that's going to lower bills on a per-request basis
In my opinion, Harness has nothing to do with energy (ie cost). They all use the same conversational requests so the efficiency is mostly in the prompt (instruction and data) as well as the chosen model.
Choose wisely.
For efficiency, providers should cache the initial prefix (ie system prompt and tools) of harnesses, not sure if they do all that.