HN Explorer

Show HN: GPT Router – Open-Source API Gateway for LLMs (OpenAI, Anthropic, etc.) (github.com)

15 points by radhakrsna ↗ HN
Hi HN,

Today, we are excited to be open-sourcing GPTRouter, an LLMOps tool we have been using internally at Writesonic for handling millions of monthly requests for our users.

Universal API for 30+ LLMs, Vision and Image Models Smart Fallbacks based on latency and uptime Automatic Retries Supports streaming

Since embracing OpenAI GPT-3 in production in 2020, we at Writesonic have been serving millions of users and faced the typical scaling pains with generative AI models:

1. Dependency on a single model risked total downtime. 2. Latency issues with models like GPT-4 affected user experience. 3. Integrating various models was tough due to different APIs and SDKs.

Early this year at Writesonic, we set out with a clear vision: to become model agnostic.

Faced with single-model limitations and diverse AI challenges, we began building GPTRouter - our bespoke solution to navigate and thrive in a multi-model AI world.

With GPTRouter's Universal API, you're the master of AI models. Swap between OpenAI, Azure, Anthropic, Replicate, Cohere & more with just one line of code. It simplifies model management to a great extent.

Downtime isn't an option. GPTRouter's Smart Fallbacks mean your service is always on. You can define a hierarchy of models for each use case. GPTRouter will constantly check for uptime/downtime, latency and other factors, and automatically fallback to the next best model with zero interruption.

Say goodbye to manual retries. GPTRouter does the heavy lifting with Automatic Retries for failed requests, keeping your AI services sharp and consistent.

GPTRouter's Edge: Universal API for seamless model switching. Smart, automatic fallbacks for continuous service. Reduced latencies for quick interactions.

Additionally, we will also be open sourcing our frontend LLMOps layer that provides a playground to test multiple models in parallel, keep a tab on the latencies for each model, track tokens and costs for each model and user all in one place.

We are looking forward to seeing how developers leverage GPTRouter in their own use cases.

Thank you!

14 comments

[ 0.30 ms ] story [ 46.2 ms ] thread
[deleted]
(comment deleted)
[deleted]
(comment deleted)
sanujsood
This is a robust LLMOps tool that brings unparalleled flexibility and reliability to AI service delivery. With its Universal API, Smart Fallbacks, Automatic Retries, and support for streaming, GPTRouter addresses critical production challenges, ensuring uninterrupted, efficient operations.
amrendrak1991
The GPT Router is an indispensable tool for anyone working with LLMs like OpenAI and Anthropic. It provides an open-source API gateway that streamlines the integration process, making it easy to harness the power of these models. With its robust features and user-friendly interface, the GPT Router is a must-have for efficient LLM utilisation.
Gupta-Anubhav12
I think this is amazing since it's open source and customizable so people can use it as a base and build new products or modify it as per their needs instead of the modification living behind an enterprise paywall it's out in wild that makes the MOAT for this
verdverm
Several early comments here looking like an organized effort to prop up the post.

Please see the posting guidelines, this is explicitly called out as something you should not do

---

Edit, @~5min, there are way more rolling in now, definitely looks like coordinated effort to inflate the post

Of 9 comments, 7 are of this type, 1 is OP, and 1 is mine

dhairyaw
Super thrilled to discover this solution - it mitigates the risk of OpenAI going down during crucial demos, offering a robust alternative.

Thank you for this solution, adding fallbacks asap in my project!! This was much needed

I also found out that you did integration within Langchain ++ :D

himanshu512
GPTRouter is a game-changing LLMOps tool, delivering unmatched flexibility and reliability in AI service delivery. Its Universal API, Smart Fallbacks, Automatic Retries, and seamless support for streaming tackle crucial production challenges head-on, guaranteeing smooth and efficient operations. A must-have for anyone aiming to elevate their AI service deployment!
hymnjack
this solves for some of the most important problems we've faced when one of the models goes for the toss, this solves for this problem really well
shreyam_maity
This seems to solve my biggest problem with my open-source project. Thanks for making this open-source.
[deleted]
(comment deleted)
Siauliai
Switching between models to address end users' requests, at the same time being able to temporarily disable some of those, which API's are down at the moment is the issue that we're addressing at the moment too. Need to dive deeper in details though, but thanks for sharing the code and knowledge base.
Amritpal001
Just gave it a go. The flexibility to switch models dynamically based on their availability and latency is impressive. It is much required for scenarios requiring high reliability. The ability to pass models' priority order adds another layer of excellence.
akbedi
This open-source tool's dynamic model switching feature offers impressive flexibility, allowing users to seamlessly switch between models and providers. The tool's abstraction layer enhances maintainability and extensibility, while clear documentation facilitates easy integration of new models and providers. Overall, this capability significantly contributes to a more adaptive and collaborative machine learning development environment. Kudos to the team for developing and open-sourcing this.