Show HN: Any-LLM – Lightweight router to access any LLM Provider (github.com)

There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve

I use Litellm Proxy, even in a dev environment via Docker, because the Usage and Logs feature greatly helps in providing visibility into LLM usage. The Caching functionality greatly helps in reducing costs for repetitive testing.

Not to be confused with AnythingLLM.

a proxy means you offload observability, filtering, caching rules, global rate limiters to a specialized piece of software - pushing this in application code means you _cannot_ do things centrally and it doesn't scale as more copies of your application code get deployed. You can bounce a single proxy server neatly vs. updating a fleet of your application server just to monkey patch some proxy functionality.

> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications

with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues)

you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.

In truth it wasn’t that hard for me to ask Claude Code to just implement the text completion API so routing wasn’t that much of a problem.

This looks awesome.

Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.

Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this.

[1] https://github.com/Airbolt-AI/airbolt

I'm excited to see this. Have been using LiteLLM but it's honestly a huge mess once you peek under the hood, and it's being developed very iteratively and not very carefully. For example. for several months recently (haven't checked in ~a month though), their Ollama structured outputs were completely botched and just straight up broken. Docs are a hot mess, etc.

I have been also working on very similar open source project for python llm abstraction layer. I needed one for my research job. I inspired from that and created one for more generic usage.

Github: https://github.com/proxai/proxai

Website: https://proxai.co/

This is awesome, will give it a try tonight.

I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?

https://xkcd.com/927/ . LiteLLM rocks !

Anything like this, but in TypeScript?

How does this differ from this project? https://github.com/simonw/llm

Crazy timing!

I shipped a similar abstraction for llms a bit over a week ago:

https://github.com/omarkamali/borgllm

pip install borgllm

I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.

What is mozilla-ai?

Seems like reputation parasitism.

Really needs a Docker image (maybe just not mentioned?) so one doesn’t have to wrestle pip and python versions.

You guys need to fact check your AI-generated blog posts:

https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to...

> One popular solution, LiteLLM, is highly valued for its wide support of different providers and modalities, making it a great choice for many developers. However, it re-implements provider interfaces rather than leveraging SDKs that are managed and released by the providers themselves. As a result, the approach can lead to compatibility issues and unexpected modifications in behavior, making it difficult to keep up with the changes happening among all the providers.

LiteLLM is rock-solid in practice. The underlying API providers announce breaking changes well in advance, and LiteLLM has never been caught out by this. LLMs will come up with hypothetical cons like this upon request.

> Lastly, proxy/gateway solutions like OpenRouter and Portkey require users to set up a hosted proxy server to act as an intermediary between their code and the LLM provider. Although this can effectively abstract away the complicated logic from the developer, it adds an extra layer of complexity and a dependency on external services, which might not be ideal for all use cases.

OpenRouter is a hosted service that provides the proxy/gateway infrastructure. Users don't "set up a hosted proxy server" themselves; they just make API calls to OpenRouter's endpoints. But older LLMs don't know what OpenRouter is and will assume it's a self-hosted proxy server.

> Another option, AISuite, was created by Andrew NG and offers a clean and modular design. However, it is not actively maintained (its last release was in December of 2024) and lacks consistent Python-typed interfaces.

Okay so you clicked the "releases" tab and saw December 2024. Next time check https://github.com/andrewyng/aisuite/commits/main/ Small, fast moving community projects like this, exllamav2, etc don't necessarily tag releases.

I've got nothing against using AI to write posts like this, but at least take the time to fact check before dumping on other people's work.

If not for the Mozilla branding, I'd have assumed this was a scam/malware - especially since it's name is so similar to Anything-LLM.