A few weeks ago, I processed a product refund with Amazon via agent. It was simple, straightforward, and surprisingly obvious that it was backed by a language model based on how it responded to my frustration about it asking tons of questions. But in the end, it processed my refund without ever connecting me with a human being.
I don't know whether Amazon relies on LLMs or SLMs for this and for similar interactions, but it makes tons of financial sense to use SLMs for narrowly scoped agents. In use cases like customer service, the intelligence behind LLMs is all wasted on the task the agents are trained for.
Wouldn't surprise me if down the road we start suggesting role-specific SLMs rather than general LLMs as both an ethics- and security-risk mitigation too.
The problem is that so many executives in charge of customer support see dollar signs with AI, and then implement the shittiest possible version of an AI chatbot that makes support interactions significantly worse. For example, I had particularly bad interactions with Doordash, Lyft, and Chipotle “support” in the past year. The chipotle one a few days ago — they gave me a completely different item than the one I ordered in the anpp, and the bot wanted me to talk to the store in person about it. Why even have an app with delivery options if that’s your answer.
If anyone working on support bots reads this, you should realize it’s very easy for a bot to tarnish your companies reputation which can add up over time. You MUST have a way to tell if interactions are deeply frustrating to the humans. Since people are used to being patient with humans, we have a higher tolerance before we become frustrated with a support person. (And support peeps know this isn’t even that much true.) Not when we know we’re talking to a bot.
Seriously, fuck all the executives who bought into chatbots for support without bothering to make it a thoughtful experience. (Which in my experience so far, is all of them. Maybe Amazon is better, but they’ve had easy refunds with a simple form for years and years!)
One could start with a large model for exploration during development, and then distill it down to a small model that covers the variety of the task and fits on a USB drive. E.g. when I use a model for gardening purposes, I could prune knowledge about other topics.
No mention of mixture-of-exports. Seems related. They do list a DeepSeek R1 distillate as an SLM. The introduction starts with sales pitch. And there's a call-to-action at the end. This seems like marketing with source references sprinkled in.
That said, I also think the "Unix" approach to ML is right. We should see more splits, however currently all these tools rely on great language comprehension. Sure, we might be able to train a model on only English and delegate translation to another model, but that will certainly lose (much needed) color. So if all of these agents will need comprehensive language understanding anyway, to be able to communicate with each other, is SLM really better than MoE?
What I'd love to "distill" out of these models is domain knowledge that is stale anyway. It's great that I can ask Claude to implement a React component, but why does the model that can do taxes so-so also try to write a React component so-so? Perhaps what's needed is a search engine to find agents. Now we're into expensive market place subscription territory, but that's probably viable for companies. It'll create a larger us-them chasm, though and the winner takes it all.
I wonder how the math turns out when we compare the energy use of local vs remote models from first principles.
A server needs energy to build it, house, power and maintain it. It is optimized for throughoutput and can be used 100% of the time. To use the server, additional energy is needed to send packets through the internet.
A local machine needs energy to build and power it. If it lives inside a person's phone or laptop, one could say housing and maintenance is free. It is optimized to have a nice form factor for personal use. It is used maybe 10% of the time or so. No energy for internet packages is needed when using the local machine.
My initial gut feeling is that the server will have way better energy efficiency when it comes to the amount of calculations it can do over its lifetime and how much energy it needs over its lifetime. But I would love to see the actual math.
I think that part of the beauty of LLMs is their versatility in so many different scenarios. When I build my agentic pipeline, I can plug in any of the major LLMs, add a prompt to it, and have it go off to do its job.
Specialized, fine-tuned models sit somewhere in between LLMs and traditional procedural code. The fine-tuning process takes time and is a risk if it goes wrong. In the meantime, the LLMs by major providers get smarter every day.
Sure enough, latency and cost are a thing. But unless you have a very specific task performed at a huge scale, you might be better off using an off-the-shelf LLM.
IMO, the paper commits an omission that undermines the thesis quite a bit: context window limitations are mentioned only once in passing (unless I missed something) and then completely ignored throughout the analysis of SLM suitability for agentic systems.
This is not a minor oversight - it's arguably, in my experience, the most prohibitive technical barrier to this vision. Consider the actual context requirements of modern agentic systems:
- Claude 4 Sonnet's system prompt alone is reportedly roughly 25k tokens for the behavioral instructions and instructions for tool use
- A typical coding agent needs: system instructions, tool definitions, current file context, broader context of the project it's working in. Additionally, you might also want to pull in documentation for any frameworks or API specs.
- You're already at 5-10k tokens of "meta" content before any actual work begins
Most SLM that can run on consumer hardware are capped at 32k or 128k contexts architecturally, but depending on what you consider a "common consumer electronic device" you'll never be able to make use of that window if you want inference at reasonable inference speeds. A 7b or 8b Model like DeepSeek-R1-Distill or Salesforce xLAM-2-8b would take 8GB of VRAM at Q4_K_M Quant with Q8_0 K/V cache at 128k context. IMO, that's not just simple consumer hardware in the sense of the broad computing market, it's enthusiast gaming hardware. Not to mention that performance degrades significantly before hitting those limits.
The "context rot" phenomenon is real: as the ratio of instructional/tool content to actual tasks content increases, models become increasingly confused, hallucinate non-existent tools or forget earlier context. If you have worked with these smaller models, you'll have experienced this firsthand - and big models like o3 or Claude 3.7/4 are not above that either.
Beyond context limitations, the paper's economic efficiency claims simply fall apart under system-level analysis. The authors present simplistic FLOP comparisons while ignoring critical inefficiencies:
- Retry tax: An LLM completing a complex task with 90% success rate might very well become 3 or 4 attempts at task completion for an SLM, each with full orchestration overhead
- Task decomposition overhead: Splitting a task that an LLM might be able to complete in one call into five SLM sub-tasks means 5x context setup, inter-task communication costs, and multiplicative error rates
- Infrastructure efficiency: Modern datacenters achieve PUE ratios near 1.1 with liquid cooling and >90% GPU utilization through batching. Consumer hardware? Gaming GPUS at 5-10% utilization, residential HVAC never designed for sustained compute, and 80-85% power conversion efficiency per device.
When you account for failed attempts, orchestration overhead and infrastructure efficiency, many "economical" SLM deployments likely consume more total energy than centralized LLM inference. It's telling that NVIDIA Research, with deep access to both datacenter and consumer GPU performance data, provides no actual system-level efficiency analysis.
For a paper positioning itself as a comprehensive analysis of SLM viability in agentic systems, sidestepping both context limitations and true system economics while making sweeping efficiency claims feels intellectually dishonest. Though, perhaps I shouldn't be surprised that NVIDIA Research concludes that running language models on both server and consumer hardware represents the optimal path forward.
I wonder if NVIDIA are worried about how concentrated their AI customer base is. The paper is more like a personal blog post than a scientific investigation.
Anyway really love the idea but many years of experience with decentralization / security / privacy projects makes me think it probably won't happen. Their description of how to incorporate SLMs at the end gives the game away: it's a description of a large, complex project that requires fairly good data science skills e.g. they just casually suggest you autoclean the data of PII then run unsupervised clustering over the results to prepare data for model fine tuning using QLoRA, and then set up an automated pipeline to do this continuously. Sure. We'll get right on that.
The history of computing is pretty simple: given a choice between spending more on hardware or more on developers, we always prefer more hardware. For NVIDIA this is a good thing modulo the fact that nobody buys their hardware directly because it's too overpowered. But that's the way they've chosen to segment the market. Given a choice between using a sledgehammer to crack a nut, or to make a custom hammer for nut cracking, we're gonna spend the money on outsourced LLMs every time.
Naturally, perhaps in future LLMs will create these SLM factories for us! If you assume software is mostly written by LLMs in future then past experience about expressed preference of software teams might not apply. But we're not there yet.
Absolutely agree-small language models are a crucial step toward scalable, efficient agentic AI. Their reduced computational footprint enables faster, on-device reasoning, which is essential for real-time decision-making in autonomous agents. Platforms like Legittmate AI are already exploring how small models can be fine-tuned for secure, task-specific workflows in legal and contract automation. With enhanced fine-tuning and optimization, SLMs are becoming increasingly capable—without the heavy cost and latency of larger models. The future of agentic AI isn't just about power-it's about precision, privacy, and portability.
Depends heavily on the complexity of your tool interface. The real power of agentic systems is not so much in their ability to select and call tools, but the expressiveness with which they can (generatively) parameterize them meaningfully in novel ways. For simple "do this or that" stuff sure, but then that could just be distilled to a deterministic workflow. There could be a world where SLMs fit into a larger system, but anything requiring sufficient creativity and exploration will still require frontier models for frontier results.
14 comments
[ 3.3 ms ] story [ 42.2 ms ] threadI don't know whether Amazon relies on LLMs or SLMs for this and for similar interactions, but it makes tons of financial sense to use SLMs for narrowly scoped agents. In use cases like customer service, the intelligence behind LLMs is all wasted on the task the agents are trained for.
Wouldn't surprise me if down the road we start suggesting role-specific SLMs rather than general LLMs as both an ethics- and security-risk mitigation too.
If anyone working on support bots reads this, you should realize it’s very easy for a bot to tarnish your companies reputation which can add up over time. You MUST have a way to tell if interactions are deeply frustrating to the humans. Since people are used to being patient with humans, we have a higher tolerance before we become frustrated with a support person. (And support peeps know this isn’t even that much true.) Not when we know we’re talking to a bot.
Seriously, fuck all the executives who bought into chatbots for support without bothering to make it a thoughtful experience. (Which in my experience so far, is all of them. Maybe Amazon is better, but they’ve had easy refunds with a simple form for years and years!)
That said, I also think the "Unix" approach to ML is right. We should see more splits, however currently all these tools rely on great language comprehension. Sure, we might be able to train a model on only English and delegate translation to another model, but that will certainly lose (much needed) color. So if all of these agents will need comprehensive language understanding anyway, to be able to communicate with each other, is SLM really better than MoE?
What I'd love to "distill" out of these models is domain knowledge that is stale anyway. It's great that I can ask Claude to implement a React component, but why does the model that can do taxes so-so also try to write a React component so-so? Perhaps what's needed is a search engine to find agents. Now we're into expensive market place subscription territory, but that's probably viable for companies. It'll create a larger us-them chasm, though and the winner takes it all.
A server needs energy to build it, house, power and maintain it. It is optimized for throughoutput and can be used 100% of the time. To use the server, additional energy is needed to send packets through the internet.
A local machine needs energy to build and power it. If it lives inside a person's phone or laptop, one could say housing and maintenance is free. It is optimized to have a nice form factor for personal use. It is used maybe 10% of the time or so. No energy for internet packages is needed when using the local machine.
My initial gut feeling is that the server will have way better energy efficiency when it comes to the amount of calculations it can do over its lifetime and how much energy it needs over its lifetime. But I would love to see the actual math.
Once you've settled down on a few well-known paths of action, wouldn't you want to freeze those paths and make it 100% predictable, for the most part?
Specialized, fine-tuned models sit somewhere in between LLMs and traditional procedural code. The fine-tuning process takes time and is a risk if it goes wrong. In the meantime, the LLMs by major providers get smarter every day.
Sure enough, latency and cost are a thing. But unless you have a very specific task performed at a huge scale, you might be better off using an off-the-shelf LLM.
This is not a minor oversight - it's arguably, in my experience, the most prohibitive technical barrier to this vision. Consider the actual context requirements of modern agentic systems:
Most SLM that can run on consumer hardware are capped at 32k or 128k contexts architecturally, but depending on what you consider a "common consumer electronic device" you'll never be able to make use of that window if you want inference at reasonable inference speeds. A 7b or 8b Model like DeepSeek-R1-Distill or Salesforce xLAM-2-8b would take 8GB of VRAM at Q4_K_M Quant with Q8_0 K/V cache at 128k context. IMO, that's not just simple consumer hardware in the sense of the broad computing market, it's enthusiast gaming hardware. Not to mention that performance degrades significantly before hitting those limits.The "context rot" phenomenon is real: as the ratio of instructional/tool content to actual tasks content increases, models become increasingly confused, hallucinate non-existent tools or forget earlier context. If you have worked with these smaller models, you'll have experienced this firsthand - and big models like o3 or Claude 3.7/4 are not above that either.
Beyond context limitations, the paper's economic efficiency claims simply fall apart under system-level analysis. The authors present simplistic FLOP comparisons while ignoring critical inefficiencies:
When you account for failed attempts, orchestration overhead and infrastructure efficiency, many "economical" SLM deployments likely consume more total energy than centralized LLM inference. It's telling that NVIDIA Research, with deep access to both datacenter and consumer GPU performance data, provides no actual system-level efficiency analysis.For a paper positioning itself as a comprehensive analysis of SLM viability in agentic systems, sidestepping both context limitations and true system economics while making sweeping efficiency claims feels intellectually dishonest. Though, perhaps I shouldn't be surprised that NVIDIA Research concludes that running language models on both server and consumer hardware represents the optimal path forward.
Anyway really love the idea but many years of experience with decentralization / security / privacy projects makes me think it probably won't happen. Their description of how to incorporate SLMs at the end gives the game away: it's a description of a large, complex project that requires fairly good data science skills e.g. they just casually suggest you autoclean the data of PII then run unsupervised clustering over the results to prepare data for model fine tuning using QLoRA, and then set up an automated pipeline to do this continuously. Sure. We'll get right on that.
The history of computing is pretty simple: given a choice between spending more on hardware or more on developers, we always prefer more hardware. For NVIDIA this is a good thing modulo the fact that nobody buys their hardware directly because it's too overpowered. But that's the way they've chosen to segment the market. Given a choice between using a sledgehammer to crack a nut, or to make a custom hammer for nut cracking, we're gonna spend the money on outsourced LLMs every time.
Naturally, perhaps in future LLMs will create these SLM factories for us! If you assume software is mostly written by LLMs in future then past experience about expressed preference of software teams might not apply. But we're not there yet.