It's an _incredibly_ important concept to understand if you have even a passing interest in LLMs. You need to understand it if you want to have any kind of mental model for how LLM-powered agents are even possible.
you> what's going on?
> It's going great --- how can I help you today?
Tool calls:
you> [json blob of available "tools": "ls", "grep", "cat"]
you> what's going on?
> [json blob selecting "ls"]
(you) presumably run "ls"
you> [json blob of "ls" output]
> [json blob selecting "cat foo.c"]
(you) dump "foo.c"
you> [json blob of "cat foo.c"]
> I can see that we're in a C project that does XYZ...
The key thing is just: tools are just a message/conversation abstraction LLMs are trained to adhere to: they know to spit out a standardized "tool call" JSON, and they know to have multi-round conversations with sets of different "tools" (whichever ones are made available to them) to build up context to answer questions with.
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
This is all true, and we've prototyped a number of these things at my current startup. You need to be pretty considered about implementing them.
For a counter-example, consider Claude Code:
- 1 long context window, with (at most) 1 sub-agent
- same tools available at all times, and to sub-agent (except: spawning a sub-sub-agent)
- Full context stays in conversation, until you hit the context window limit. Compaction is automatic but extremely expensive. Quality absolutely takes a dive until everything is re-established.
- Deterministic lookup of content. Claude reads files with tools, not includes random chunks from RAG cosine similarity.
I could go on. In my experience, if you're going to use these techniques 1) maybe don't and 2) turn up the determinism to 11. Get really specific about _how_ you're going to use, and why, in a specific case.
For example, we're working on code migrations [0]. We have a tool that reads changelogs, migration guides, and OSS source. Those can be verbose, so they blow the context window on even 200k models. But we're not just randomly deleting things out of the "plan my migration" context, we're exposing a tool that deliberately lets the model pull out the breaking changes. This is "Context Summarization," but before using it, we had to figure out that _those_ bits were breaking the context, _then_ summarizing them. All our attempts at generically pre-summarizing content just resulted in poor performance because we were hiding information from the agent.
This isn't going to be much of a problem for long. I'm wrapping up on an agent context manager that gives effectively infinite context while producing ~77% better results than naive vector+BM25 baseline on my benchmark suite.
> When prompting DeepSeek-v3, the team found that selecting the the right tools becomes critical when you have more than 30 tools. Above 30, the descriptions of the tools begin to overlap, creating confusion. Beyond 100 tools, the model was virtually guaranteed to fail their test. Using RAG techniques to select less than 30 tools yielded dramatically shorter prompts and resulted in as much as 3x better tool selection accuracy.
> For smaller models, the problems begin long before we hit 30 tools. One paper we touched on last post, “Less is More,” demonstrated that Llama 3.1 8b fails a benchmark when given 46 tools, but succeeds when given only 19 tools. The issue is context confusion, not context window limitaions.
High number of tools is a bit of a "smell" to me and often makes me wonder if the agent doesn't have too much responsibility. A bit like a method with so many parameters, it can do almost anything.
Have folks had success with agents like that? I found the fewer tools the better, e.g. <10 "ballpark".
we have success with 39 but we're introducing more focused agents and a smart router because we see the writing on the wall among other things (benefits)
13 comments
[ 2.1 ms ] story [ 30.8 ms ] threadHowever it uses various terms that I'm not sure of the definition for.
E.g. the word "Tool".
How can I learn about the definitions of these words? Will reading prior articles in the series help? Please advise.
https://youtu.be/owDd1CJ17uQ?si=Z2bldI8IssG7rGON&t=1330
It's an _incredibly_ important concept to understand if you have even a passing interest in LLMs. You need to understand it if you want to have any kind of mental model for how LLM-powered agents are even possible.
That's the whole thing.
Exactly, the issue is at:
(also 'from-font' would break it.)It's clearly not intentional - it's a glitch, probably in the font.
https://news.ycombinator.com/newsguidelines.html
For a counter-example, consider Claude Code:
- 1 long context window, with (at most) 1 sub-agent
- same tools available at all times, and to sub-agent (except: spawning a sub-sub-agent)
- Full context stays in conversation, until you hit the context window limit. Compaction is automatic but extremely expensive. Quality absolutely takes a dive until everything is re-established.
- Deterministic lookup of content. Claude reads files with tools, not includes random chunks from RAG cosine similarity.
I could go on. In my experience, if you're going to use these techniques 1) maybe don't and 2) turn up the determinism to 11. Get really specific about _how_ you're going to use, and why, in a specific case.
For example, we're working on code migrations [0]. We have a tool that reads changelogs, migration guides, and OSS source. Those can be verbose, so they blow the context window on even 200k models. But we're not just randomly deleting things out of the "plan my migration" context, we're exposing a tool that deliberately lets the model pull out the breaking changes. This is "Context Summarization," but before using it, we had to figure out that _those_ bits were breaking the context, _then_ summarizing them. All our attempts at generically pre-summarizing content just resulted in poor performance because we were hiding information from the agent.
[0] https://tern.sh
> For smaller models, the problems begin long before we hit 30 tools. One paper we touched on last post, “Less is More,” demonstrated that Llama 3.1 8b fails a benchmark when given 46 tools, but succeeds when given only 19 tools. The issue is context confusion, not context window limitaions.
High number of tools is a bit of a "smell" to me and often makes me wonder if the agent doesn't have too much responsibility. A bit like a method with so many parameters, it can do almost anything.
Have folks had success with agents like that? I found the fewer tools the better, e.g. <10 "ballpark".