HN Explorer

Show HN: Frugon – Find which LLM calls a cheaper model could handle (local, MIT) (github.com)

67 points by jarodrh ↗ HN
I started leaning in on AI heavily this year, as I wanted to get more done autonomously, but then my token usage climbed dramatically to the point where my weekly quota would run out before the end of the week, sometimes a couple of days into the week.

I realised I had to do something about it else I'd have to double my spend. So I decided to start tracking my cost per task type. This revealed that a lot of my spend went to searches/scans or simple things like scouting tasks.

I then decided to turn this into a simple CLI tool that can be used to read your OpenAI-style logs locally, and analyze the cost and compare this spend to other models, then show you how much you could potentially save by switching those calls to a cheaper model.

When you run analyze you get an offline estimate priced against LiteLLM and gated by LMArena tiers. The general savings bands come from the research published by RouteLLM; but you can confirm this yourself using 2 commands --measure (shows the prompt-response output side by side) and --judge (a model chosen to do the comparisons). These send a sample of the prompts from the logs to the candidate models - either the default choice or set by you. This call goes directly to the model provider (never through me) as any normal LLM call would, and the response is shown and judged to either be better or worse or a tie.

It's deliberately small, because I tend to over complicate/think things sometimes: analyze + capture + a few commands, doing three jobs. Cost, quality visibility, routing recommendation.

Nothing is hosted. capture is an optional local proxy on your own machine, and there's no endpoint in the path of your data. You can confirm this by checking the source.

I included a demo so you can check out the output. It has a synthetic 56k call log (a month's worth) showing how costs can drop from $549.46 to $343.91 a month. A 37.4% saving.

Try it:

  uvx frugon analyze --demo
or

  uv tool install frugon
Then point it at your own logs.

All feedback is welcome, especially any on the routing/quality logic, or anything else, good or bad.

20 comments

[ 2.8 ms ] story [ 30.4 ms ] thread
[deleted]
(comment deleted)
cyanydeez
this would be more interesting as a local LLM anlysis; throw out all the costs, and figure out primary-subagent model architecture, and maximize token generation and prefill.

I don't see how anyone can operationalize this information.

jarodrh
That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a cost perspective as opposed to a raw token generation/prefill perspective. The mindset can be applied to both sides, but Frugon prices the cloud side.

The end goal is essentially the same and your mind went to that point, "primary-subagent model architecture". This is what the tool helps you figure out. It's not there to hold your hand and explain what your architecture should be, as I wanted it to be a small simple tool that would give the user insight into triggering your exact thought process. The thought process of breaking down their tasks by type, mapping that to individual models regardless of app or dev tool, regardless of cloud or local (the thought process transfers). It shows the user that they could route a portion of their calls to a cheaper model. It's then up to the user to understand the task type and point those calls to different models.

To directly answer the operationalization statement, it depends on what the call log is from (app/harness).

If harness, then the direction would be to pin models per role per task type; the routing recommendation maps directly to that, and measure/judge allows you to verify this before switching.

If app, then this is a similar shape as above, where you would then categorise and pin those calls identified in the recommendation to the model recommended, and as above, measure/judge allows you to verify before switching.

You're proposing an optimisation for throughput whereas Frugon is for spend. Same issue, just different lens.

XUEYANZ
Haha. somehow i just love the naming. it just makes sense :D
jarodrh
Thanks! I love a good meaningful name

Frugal + on

:P

XUEYANZ
Exactly. That's my first impression
isadubois
This looks super clean. I'm curious about the --judge command. How does it evaluate if the cheaper model's response is a "tie" or acceptable? Is it using a specific LLM-as-a-judge prompt template?
[dead] tokoi
[flagged]
ipanditshashi
Useful but how it compares with other providers model
pauldavis
I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
jarodrh
Thanks. Yes, local models are gaining a lot of traction.

The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.

Noted: "response time as a new dimension of judging" - Added to the roadmap.

Try it and let me know if you hit a wall.

santiago-pl
If you experience any issues with LiteLLM, you may try GoModel - the AI Gateway I'm working on. It consumes ~60x less resources and is more reliable :)
indigodaddy
What we need is an AI gateway/router that will actually first analyze the input tokens and then decide what model to use. If it's so simple that a dirt cheap qwen 3.5 flash or whatever will be fine, then it chooses that. If it deems we need GPT 5.6, then it uses that, etc. does anything like this already exist?
westurner
Evals and OpenInference (OpenTelemetry) might be useful.

Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well.

Is this the cost to converge problem, the minimize cost to converge upon sufficiently low error problem, or the minimize cost and error problem?

EA methods: mutation, crossover, selection

Gradient descent as a mutation, crossover, and selection pattern; back up when the error/cost stops decreasing for too long and try a different branch.

A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

mujia
A cheaper model can look fine per request, but one weak answer may create another call or a human review step. That seems like the hardest cost to capture.