We built a unified API with a large surface area and ran into a problem when building our MCP server: tool definitions alone burned 50,000+ tokens before the agent touched a single user message.
The fix that worked for us was giving agents a CLI instead. ~80 tokens in the system prompt, progressive discovery through --help, and permission enforcement baked into the binary rather than prompts.
The post covers the benchmarks (Scalekit's 75-run comparison showed 4-32x token overhead for MCP vs CLI), the architecture, and an honest section on where CLIs fall short (streaming, delegated auth, distribution).
What I've done with my MCPs is turning them into a CLI, except there's still an MCP server that only has the instructions to tell the the agent about the CLI.[1] Claude and GLM-5 seem to have no problems with it.
As a bonus, the entire thing now works as a plain old CLI too - which it honestly should have from the beginning.
Fair point on tool search. Claude Code and Codex do have it.
But tool search is solving the symptom, not the cause. You still pay the per-tool token cost for every tool the search returns. And you've added a search step (with its own latency and token cost) before every tool call.
With a CLI, the agent runs `--help` and gets 50-200 tokens of exactly what it needs. No search index, no ranking, no middleware. The binary is the registry.
Tool search makes MCP workable. CLIs make the search unnecessary.
Why not use skills? They follow a three-tier loading approach, and you can stick an MCP as part of the toolset for the skills, so it will only load it when the skill is selected.
I'm getting tired of everyone saying "MCP is dead, use CLIs!".
Yes, MCP eats up context windows, but agents can also be smarter about how they load the MCP context in the first place, using similar strategy to skills.
The problem with tossing it out entirely is that it leaves a lot more questions for handling security.
When using skills, there's no implicit way to be able to apply policies in the sane way across many different servers.
MCP gives us a registry such that we can enforce MCP chain policies, i.e. no doing web search after viewing financials.
Doing the same with skills is not possible in a programatic and deterministic way.
There needs to be a middle ground instead of throwing out MCP entirely.
It is a weird trend. I see the appeal of Skills over MCP when you are just a solo dev doing your work. MCP is incredibly useful in an organization context when you need to add controls and process. Both are useful. I feel like the anti-MCP push is coming from people who don't need to work in a large org.
I agree, and it's context-dependent when to use what (the author mentions use cases for other solutions). I'm glad there are multiple solutions to choose from.
Skills are just prompts, so policy doesn't apply there. MCP isn't giving you any special policy control there, it's just a capability border. You could do the same thing with a service mesh or any other capability compartmentalization technique.
The only value in MCP is that it's intended "for agents" and it has traction.
> Yes, MCP eats up context windows, but agents can also be smarter about how they load the MCP context in the first place, using similar strategy to skills.
I have been keeping an eye on MCP context usage with Claude Code's /context command.
When I ran it a couple months ago, supabase used 13.2k tokens all the time, with the search_docs tool using 8k! So, I disabled that tool in my config.
I just ran /context now, and when not being used it uses only ~300 tokens.
I have a question. Does anyone know a good way to benchmark actual MCP context usage in Claude Code now? I just tried a few different things and none of them worked.
I feel like I don't fully understand mcp. I've done research on it but I definitely couldn't explain it. I get lost on the fact that to my knowledge it's a server with API endpoints that are well defined into a json schema then sent the to LLM and the LLM parses that and decides which endpoints to hit (I'm aware some llms use smart calling now so they load the tool name and description but nothing else until it's called). How exactly are you doing the process of stopping the LLM from using web search after it hits a certain endpoint in your MCP server? Or is this referring strictly to when you own the whole workflow where you can then deny websearch capabilities on the next LLM step?
Are there any good docs youve liked to learn about it, or good open source projects you used to get familiar? I would like to learn more
> I'm getting tired of everyone saying "MCP is dead, use CLIs!".
The people saying this and attacking it should first agree about the question.
Are you combining a few tools in the training set into a logical unit to make a cohesive tool-suite, say for reverse engineering or network-debugging? Low stakes for errors, not much on-going development? Great, you just need a thin layer of intelligence on top of stack-overflow and blog-posts, and CLI will probably do it.
Are you trying to weld together basically an AI front-end for an existing internal library or service? Is it something complex enough that you need to scale out and have modular access to? Is it already something you need to deploy/develop/test independently? Oops, there's nothing quite like that in the training set, and you probably want some guarantees. You need a schema, obviously. You can sort of jam that into prompts and prayers, hope for the best with skills, skip validation and risk annotations being ignored, trust that future opaque model-change will be backwards compatible with how skills are even selected/dispatched. Or.. you can use MCP.
Advocating really hard for one or the other in general is just kind of naive.
IMO if you want a metadata registry of how actions work so you can make complicated, fragile, ACL rule systems of actions, then make that. That doesn't need to be loaded into a context window to make that work and can be expanded to general API usage, tool usage, cli usage, and so on. You can load a gh cli metadata description system and so on.
MCPs are clunky, difficult to work with and token inefficient and security orgs often have bad incentive design to mostly ignore what the business and devs need to actually do their job, leading to "endpoint management" systems that eat half the system resources and a lot of fig leaf security theatre to systematically disable whatever those systems are doing so people can do their job in an IT equivalent that feels like the TSA.
Thank god we moving away from giving security orgs these fragile tools to attach ball and chains to everyone.
> MCP gives us a registry such that we can enforce MCP chain policies
Do you have some more info on it?
looking up "registry" in the mcp spec will just describe a centrally hosted, npm-like package registry[^1]
[^1]: The MCP Registry is the official centralized metadata repository for publicly accessible MCP servers, backed by major trusted contributors to the MCP ecosystem such as Anthropic, GitHub, PulseMCP, and Microsoft.
CLIs are great for some applications! But 'progressive disclosure' means more mistakes to be corrected and more round trips to the model - every time[1] you use the tool in a new thread. You're trading latency for lower cost/more free context. That might be great! But it might not be, and the opposite trade (more money/less context for lower latency) makes a lot of sense for some applications. esp. if the 'more money' part can be amortized over lots of users by keeping the tool definitions block cached.
[1]: one might say 'of course you can just add details about the CLI to the prompt' ... which reinvents MCP in an ad hoc underspecified non-portable mode in your prompt.
One of the MCP Core Maintainers here, so take this with a boulder of salt if you're skeptical of my biases.
The debate around "MCP vs. CLI" is somewhat pointless to me personally. Use whatever gets the job done. MCP is much more than just tool calling - it also happens to provide a set of consistent rails for an agent to follow. Besides, we as developers often forget that the things we build are also consumed by non-technical folks - I have no desire to teach my parents to install random CLIs to get things done instead of plugging a URI to a hosted MCP server with a well-defined impact radius. The entire security posture of "Install this CLI with access to everything on your box" terrifies me.
The context window argument is also an agent harness challenge more than anything else - modern MCP clients do smart tool search that obviates the entire "I am sending the full list of tools back and forth" mode of operation. At this point it's just a trope that is repeated from blog post to blog post. This blog post too alludes to this and talks about the need for infrastructure to make it work, but it just isn't the case. It's a pattern that's being adopted broadly as we speak.
> modern MCP clients do smart tool search that obviates the entire "I am sending the full list of tools back and forth" mode of operation
This has always surprised me as this always comes up in MCP discussions.
To me, it just seem like a matter of updating the protocol to not have that context hungry behaviour. Doesn't seem like an insurmountable problem technically.
Glad you say it has already been addressed. Was the protocol itself updated to reflect that? Or are you just referring to off-spec implementations?
If you don't change your approach but just use CLI "intead of" MCP, you'll end up with a new spin on the same problems. The guardrails MCP provides (identity, entitlement, multi-principal trust boundaries) still need to exist somewhere.
The industry is talking in circles here. All you need is "composability".
UNIX solved this with files and pipes for data, and processes for compute.
AI agents are solving this this with sub-agents for data, and "code execution" for compute.
The UNIX approach is both technically correct and elegant, and what I strongly favor too.
The agent + MCP approach is getting there. But not every harness has sub-agents, or their invocation is non-deterministic, which is where "MCP context bloat" happens.
What's interesting to me is that while it was obvious to all of us who came to think in the Unix Way, that insofar as composability, usage discoverability, and gobs of documentation in posts and man pages that are hugely represented in training corpora for LLMs, that the CLI is a great fit for LLM tool use, it seems only a recent trend to acknowledge this (and also the next hype wave, perhaps.)
Also interesting that while the big vendors are following this trend and are now trying to take a lead in it, they still suggest things like "but use a JSON schema" (the linked article does a bit of the same - acknowledging that incremental learning via `--help` is useful AND can be token-conserving (exception being that if they already "know" the correct pattern, they wouldn't need to use tokens to learn it, so there is a potential trade-off), they are also suggesting that LLMs would prefer to receive argument knowledge in json rather than in plain language, even though the entire point of an LLM is for understand and create plain language. Seemed dubious to me, and a part of me wondered if that advice may be nonsense motivated by desire to sell more token use. I'm only partially kidding and I'm still dubious of the efficacy.
* Here's a TL;DR for anyone who wants to skip the rest of this long message: I ran an LLM CLI eval in the form of a constructed CTF. Results and methodology are in the two links in the section linked:
https://github.com/scottvr/jelp?tab=readme-ov-file#what-else
Anyhow... I had been experimenting with the idea of having --help output json when used by a machine, and came up with a simple module that exposes `--help` content as json, simply by adding a `--jelp` argument to any tool that already uses argparse.
In the process, I started testing, to see if all this extra machine-readable content actually improved performance, what it did to token use, etc. While I was building out test, trying to settle on legitimate and fair ways to come to valid conclusions, I learned of the OpenCLI schema draft, so I altered my `jelp` output to fit that schema, and set about documenting the things I found lacking from the schema draft, meanwhile settling to include these arg-related items as metadata in the output.
I'll get to the point. I just finished cleaning the output up enough to put it in a public repo, because my intent is to share my findings with the OpemCLI folks, in hopes that they'll consider the gaps in their schema compared to what's commonly in use, but at the same time, what came as a secondary thought in service of this little tool I called "jelp", is a benchmarking harness (and the first publishable results from it), the to me, are quite interesting and I would be happy if others found it to be and added to the existing test results with additional runs, models, or ideas for the harness, or criticism about the validity of the method, etc.
The evaluation harness uses constructed CLI fixtures arranged as little CLI CTF's, where the LLMs demonstrate their ability to use an unknown CLI be capturing a "flag" that they'll need to discover by using the usage help, and a trail of learned arguments.
My findings at first confirmed my intuitions, which was disappointing but unsurprising. When testing with GPT-4.1-mini, no manner of forcing them to receive info about the CLI via json was more effective than just letting them use the human-friendly plain English output of --help, and in all cases the JSON versions burned more tokens. I was able to elicit better performance by some measurements from 5.1-mini, but again the tradeoff was higher token burn.
I'll link straight to the part of the README that shows one table of results, and contains links to the LLM CLI CTF part of the repo, as well as the generated report after the phase-1 runs; all the code to reproduce or run your ow...
> Limit integrations → agent can only talk to a few services
The idea that people see this as one horn of a trilemma instead of just good practice is a bit strange. Who would complain that every import isn't a star-import? Bring in what you need at first, then load new things dynamically with good semantics for cascade / drill-down. Let's maybe abandon simple classics like namespacing and the unix philsophy for the kitchen-sink approach after the kitchen-sink thing is shown to work.
71 comments
[ 0.33 ms ] story [ 103 ms ] threadThe fix that worked for us was giving agents a CLI instead. ~80 tokens in the system prompt, progressive discovery through --help, and permission enforcement baked into the binary rather than prompts.
The post covers the benchmarks (Scalekit's 75-run comparison showed 4-32x token overhead for MCP vs CLI), the architecture, and an honest section on where CLIs fall short (streaming, delegated auth, distribution).
As a bonus, the entire thing now works as a plain old CLI too - which it honestly should have from the beginning.
[1]: https://github.com/jcdickinson/ferrisfetch/blob/main/cmd/mcp...
Very interesting topic, but this LLM structure is instant anthema I just have to stop reading once I smell it.
The major harnesses like Claude Code + Codex have had tool search for months now.
But tool search is solving the symptom, not the cause. You still pay the per-tool token cost for every tool the search returns. And you've added a search step (with its own latency and token cost) before every tool call.
With a CLI, the agent runs `--help` and gets 50-200 tokens of exactly what it needs. No search index, no ranking, no middleware. The binary is the registry.
Tool search makes MCP workable. CLIs make the search unnecessary.
See the progressive disclosure section in the skills docs: https://agentskills.io/what-are-skills
Yes, MCP eats up context windows, but agents can also be smarter about how they load the MCP context in the first place, using similar strategy to skills.
The problem with tossing it out entirely is that it leaves a lot more questions for handling security.
When using skills, there's no implicit way to be able to apply policies in the sane way across many different servers.
MCP gives us a registry such that we can enforce MCP chain policies, i.e. no doing web search after viewing financials.
Doing the same with skills is not possible in a programatic and deterministic way.
There needs to be a middle ground instead of throwing out MCP entirely.
The only value in MCP is that it's intended "for agents" and it has traction.
I have been keeping an eye on MCP context usage with Claude Code's /context command.
When I ran it a couple months ago, supabase used 13.2k tokens all the time, with the search_docs tool using 8k! So, I disabled that tool in my config.
I just ran /context now, and when not being used it uses only ~300 tokens.
I have a question. Does anyone know a good way to benchmark actual MCP context usage in Claude Code now? I just tried a few different things and none of them worked.
Are there any good docs youve liked to learn about it, or good open source projects you used to get familiar? I would like to learn more
Here's a longer piece on why the trust boundary has to live at the runtime level, not the interface level, and what that means for MCP's actual job: https://forestmars.substack.com/p/twilight-of-the-mcp-idols
The people saying this and attacking it should first agree about the question.
Are you combining a few tools in the training set into a logical unit to make a cohesive tool-suite, say for reverse engineering or network-debugging? Low stakes for errors, not much on-going development? Great, you just need a thin layer of intelligence on top of stack-overflow and blog-posts, and CLI will probably do it.
Are you trying to weld together basically an AI front-end for an existing internal library or service? Is it something complex enough that you need to scale out and have modular access to? Is it already something you need to deploy/develop/test independently? Oops, there's nothing quite like that in the training set, and you probably want some guarantees. You need a schema, obviously. You can sort of jam that into prompts and prayers, hope for the best with skills, skip validation and risk annotations being ignored, trust that future opaque model-change will be backwards compatible with how skills are even selected/dispatched. Or.. you can use MCP.
Advocating really hard for one or the other in general is just kind of naive.
MCPs are clunky, difficult to work with and token inefficient and security orgs often have bad incentive design to mostly ignore what the business and devs need to actually do their job, leading to "endpoint management" systems that eat half the system resources and a lot of fig leaf security theatre to systematically disable whatever those systems are doing so people can do their job in an IT equivalent that feels like the TSA.
Thank god we moving away from giving security orgs these fragile tools to attach ball and chains to everyone.
Do you have some more info on it?
looking up "registry" in the mcp spec will just describe a centrally hosted, npm-like package registry[^1]
[^1]: The MCP Registry is the official centralized metadata repository for publicly accessible MCP servers, backed by major trusted contributors to the MCP ecosystem such as Anthropic, GitHub, PulseMCP, and Microsoft.
[1]: one might say 'of course you can just add details about the CLI to the prompt' ... which reinvents MCP in an ad hoc underspecified non-portable mode in your prompt.
This might be a complete non issue in 6 months.
The debate around "MCP vs. CLI" is somewhat pointless to me personally. Use whatever gets the job done. MCP is much more than just tool calling - it also happens to provide a set of consistent rails for an agent to follow. Besides, we as developers often forget that the things we build are also consumed by non-technical folks - I have no desire to teach my parents to install random CLIs to get things done instead of plugging a URI to a hosted MCP server with a well-defined impact radius. The entire security posture of "Install this CLI with access to everything on your box" terrifies me.
The context window argument is also an agent harness challenge more than anything else - modern MCP clients do smart tool search that obviates the entire "I am sending the full list of tools back and forth" mode of operation. At this point it's just a trope that is repeated from blog post to blog post. This blog post too alludes to this and talks about the need for infrastructure to make it work, but it just isn't the case. It's a pattern that's being adopted broadly as we speak.
This has always surprised me as this always comes up in MCP discussions. To me, it just seem like a matter of updating the protocol to not have that context hungry behaviour. Doesn't seem like an insurmountable problem technically.
Glad you say it has already been addressed. Was the protocol itself updated to reflect that? Or are you just referring to off-spec implementations?
If you don't change your approach but just use CLI "intead of" MCP, you'll end up with a new spin on the same problems. The guardrails MCP provides (identity, entitlement, multi-principal trust boundaries) still need to exist somewhere.
https://forestmars.substack.com/p/twilight-of-the-mcp-idols
UNIX solved this with files and pipes for data, and processes for compute.
AI agents are solving this this with sub-agents for data, and "code execution" for compute.
The UNIX approach is both technically correct and elegant, and what I strongly favor too.
The agent + MCP approach is getting there. But not every harness has sub-agents, or their invocation is non-deterministic, which is where "MCP context bloat" happens.
Source: building an small business agent at https://housecat.com/.
We do have APIs wrapped in MCP. But we only give the agent BASH, an CLI wrapper for the MCPs, and the ability to write code, and works great.
"It's a UNIX system! I know this!"
https://linuxtoaster.com/blog/gradientdescentforcode.html
Also interesting that while the big vendors are following this trend and are now trying to take a lead in it, they still suggest things like "but use a JSON schema" (the linked article does a bit of the same - acknowledging that incremental learning via `--help` is useful AND can be token-conserving (exception being that if they already "know" the correct pattern, they wouldn't need to use tokens to learn it, so there is a potential trade-off), they are also suggesting that LLMs would prefer to receive argument knowledge in json rather than in plain language, even though the entire point of an LLM is for understand and create plain language. Seemed dubious to me, and a part of me wondered if that advice may be nonsense motivated by desire to sell more token use. I'm only partially kidding and I'm still dubious of the efficacy.
* Here's a TL;DR for anyone who wants to skip the rest of this long message: I ran an LLM CLI eval in the form of a constructed CTF. Results and methodology are in the two links in the section linked: https://github.com/scottvr/jelp?tab=readme-ov-file#what-else
Anyhow... I had been experimenting with the idea of having --help output json when used by a machine, and came up with a simple module that exposes `--help` content as json, simply by adding a `--jelp` argument to any tool that already uses argparse.
In the process, I started testing, to see if all this extra machine-readable content actually improved performance, what it did to token use, etc. While I was building out test, trying to settle on legitimate and fair ways to come to valid conclusions, I learned of the OpenCLI schema draft, so I altered my `jelp` output to fit that schema, and set about documenting the things I found lacking from the schema draft, meanwhile settling to include these arg-related items as metadata in the output.
I'll get to the point. I just finished cleaning the output up enough to put it in a public repo, because my intent is to share my findings with the OpemCLI folks, in hopes that they'll consider the gaps in their schema compared to what's commonly in use, but at the same time, what came as a secondary thought in service of this little tool I called "jelp", is a benchmarking harness (and the first publishable results from it), the to me, are quite interesting and I would be happy if others found it to be and added to the existing test results with additional runs, models, or ideas for the harness, or criticism about the validity of the method, etc.
The evaluation harness uses constructed CLI fixtures arranged as little CLI CTF's, where the LLMs demonstrate their ability to use an unknown CLI be capturing a "flag" that they'll need to discover by using the usage help, and a trail of learned arguments.
My findings at first confirmed my intuitions, which was disappointing but unsurprising. When testing with GPT-4.1-mini, no manner of forcing them to receive info about the CLI via json was more effective than just letting them use the human-friendly plain English output of --help, and in all cases the JSON versions burned more tokens. I was able to elicit better performance by some measurements from 5.1-mini, but again the tradeoff was higher token burn.
I'll link straight to the part of the README that shows one table of results, and contains links to the LLM CLI CTF part of the repo, as well as the generated report after the phase-1 runs; all the code to reproduce or run your ow...
The idea that people see this as one horn of a trilemma instead of just good practice is a bit strange. Who would complain that every import isn't a star-import? Bring in what you need at first, then load new things dynamically with good semantics for cascade / drill-down. Let's maybe abandon simple classics like namespacing and the unix philsophy for the kitchen-sink approach after the kitchen-sink thing is shown to work.