Show HN: We made glhf.chat – run almost any open-source LLM, including 405B (glhf.chat)
Hey HN!
We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant.
Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and it’ll work (as long as vLLM supports the base model architecture, we’ll run anything up to ~640GB of VRAM, give or take a little for some overhead buffer).
Large models will take a few minutes to boot, but if a bunch of people are trying to use the same model, it might already be loaded and not need boot time at all. The Llama-3-70b finetunes are especially nice, since they’re basically souped-up versions of the 8b finetunes a lot of people like to run locally but don’t have the VRAM for. We’re expecting the Llama-3.1 finetunes to be pretty great too once they start getting released.
There are some caveats for now — for example, while we support the Deepseek V2 architecture, we actually can only run their smaller “Lite” models due to some underlying NVLink limitations (though we’re working on it). But for the most part if vLLM supports it, we should too!
We figured Llama-3.1-405B Launch Day was a good day to launch ourselves too — let us know in the comments if there’s anything you want us to support, or if you run into any issues. I know it’s not “local” Llama, but, well, that’s a lot of GPUs…
97 comments
[ 3.1 ms ] story [ 194 ms ] threadWhat concerns me more is the lack of any privacy policy or anything about training on inputs/outputs and much more.
The lack of a system prompt seems weird too.
How does this compare to OpenRouter.ai?
https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
Yeah we should add a privacy policy / ToS: good point.
(let's talk about access to GPUs with a lot more vram and no nvlink issues... mi300x, email in my profile)
Your service looks interesting, but I think you need to be more transparent about your infrastructure. To which "inference providers" do you proxy to, and when? Who is hosting the GPU clusters?
Also, a privacy policy and ToS document are pretty important, even at this stage.
How this compares to OpenRouter is something I'd like to know as well.
https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
1. If Together.ai has the model, we proxy to them, since they're faster than us. We might switch to Fireworks for the Llama-3.1 models since they offer them at lower cost; in general that's why I didn't specify the inference providers, since we'll probably change and optimize that quite a bit. (Also would be interesting to try using Groq, since they're so fast.)
2. If the model isn't hosted somewhere else, e.g. a lot of the Llama 3 finetunes, we run the models on our own GPU clusters hosted in Fly.io. This will probably change in the future as well, since some models would really benefit from NVLink (which Fly doesn't support currently).
2) ...
3) Profit
We do have API support! We expose an OpenAI compatible API. You can see details when logged in at https://glhf.chat/users/settings/api
Just like our web UI it supports feeding in any huggingface user/repo.
(Also available via the user menu)
Let us know if you have any questions/feedback!
Great point. Right now we don't log or store any chat messages for the API (only what models people are choosing to run). We do store messages for the web UI chat history and only share it with inference providers (currently together.ai) per request for popular models, but I know some hand-waved details from an HN comment doesn't suffice.
We'll get on that ASAP. :)
what version llama are you
I am an instance of Llama 2, an artificial intelligence model developed by Meta AI.
how many parameters
I am a Llama 2 model with 7 billion parameters.
Attempt 2
which llama are you exactly
I am a variant of the LLaMA model, specifically LLaMA 1.5B.
Here's a brief overview of the different LLaMA models:
LLaMA 7B: This is the largest and most powerful LLaMA model, with 7 billion parameters. It's the most advanced and capable model in the LLaMA family.
LLaMA 3B: This model has 3 billion parameters and is a smaller but still very capable version of the LLaMA model.
LLaMA 1.5B: This is the model I'm based on, with 1.5 billion parameters. It's a smaller and more efficient version of the LLaMA model, but still very knowledgeable and capable.
LLaMA 650M: This is the smallest LLaMA model, with 650 million parameters. It's a more lightweight and efficient version of the model, but still useful for many tasks.
is 3.1 this bad at hallucinations? or are you just serving something else
This just keeps on giving
when does your training data end
" My training data is a massive corpus of text that was available to me up to a certain point in time. My training data is cut off at December 2021."
* For always-on proxied models, we're dependent on the inference provider. Typically we use Together AI, but for Llama 3.1 we found some show-stopping context length related bugs that caused us to switch to Fireworks.
Since we proxy 405B to Fireworks, that one runs at FP8 (that's what Fireworks uses). This may change in the future and in general we prefer to match the underlying repo quantization when possible; proxying is a bit of a hack in order to keep more models warmed up and reduce boot times (and to run 405B for now, although we're working on getting larger clusters and supporting distributed multi-node inference). As we grow, hopefully we'll be able to reduce our dependency on third party proxies.
I'm also curious to try running models on AMD, since that would allow for much larger single-node models — for example, an MI300X has 192GB per card, so 8x of those would run 405B at fp16. Obviously there are some support issues with AMD so TBD how well it'll work but it's an interesting option.
> Who are you?
In my own machine without a system prompt and it said:
> I am a language model developed through a collaboration between Mistral AI and NVIDIA.
Often times this information will be included in the instruct fine tuning.
The same from qwen2:7b-instruct-q8_0:
> I am a large language model created by Alibaba Cloud. I'm called Qwen.
All I did was change the model and re-run the prompt in Open WebUI, so if there was a system prompt, it would have been used for both.
It'd be like asking a child who was never spoken to before their name. How would the child know until someone told it?
Probably need to know more about you. Main concerns are "can this be used for production" and "will you be around in a year" and "will you likely have 429s if you get too popular?" and of course likely pricing.
We're working on fleshing out ToS, privacy policy, and company specifics, but just to answer your first question, I'm Billy Cao, an ex-Google eng, and Matt Baker is ex-Airbnb, ex-Meta.
Re: concerns, our infra will scale relatively well (several qps per model, probably), but we're still in the stages of fleshing things out and getting feedback. :)
Feel free to drop us a line at hi@glhf.chat if you wanted to chat specifics!
- Billy
Nothing specific... just wanted to get the feel. Out of interest are you bare metal or using AWS or something? Or both?
And in Google Vertex its free at the moment!
We currently use vllm under the hood and vllm doesn't support Codestral (yet). We're working on expanding our model support. Hence (almost) any model.
Thanks for testing! :)
https://github.com/vllm-project/vllm/issues/6479
- Billy :)
Using money that is yours should never be illegal. Prosecute the bad people for the bad thing they did in the first place to get the money / the bad thing that someone paid them money for.
Are you saying that Anti money laundering rules should be scrapped?
How do you expect to do that, without following the money?
If someone shows up with a suitcase full of money, it's extremely unlikely that they earned this money in a legal way and paid taxes.
Same thing with crypto. Theoretically it's possible that people use it for legal means, in practice 99% of people do not.
> Theoretically it's possible that people use it for legal means, in practice 99% of people do not.
This is a completely made up statistic and it is showing your bias and/or ignorance about the topic. The lower estimates from industry sources like Chainalysis (0.15-0.62%) contrast sharply with higher academic estimates (23-46%) because the latter tend to include illicit activities that happen off chain but get "washed" on-chain which explains their own huge range of estimation... it is hard to quantify but nobody serious ever came close to 99%.
Something for which there are estimates close to 90% is the volume of transactions happening on centralized exchanges, and since these are required pretty much everywhere to follow AML procedures, just like traditional banks (sometimes even more intrusive than banks), it means it is just as easy, if not easier, to prosecute criminals who would use these... with the added bonus of having a public ledger with records of their activity on-chain.
[1] https://anchor.fm/s/9a92cef4/podcast/play/84999897/https%3A%...
Crypto is not money. Money is coupled to a value; crypto is as much a currency as paintings are: completely arbitrary.
Then top it off by claiming someone would want a currency that deflates 20,000%?
Of course it's way more complex than that, but that's the basic difference between real-world money and crypto currencies.
Given. Exchange.
If you and I agree that we only accept little tin circles; What's the difference? When you say money, what are you referring to? Just the US bank note? Some brass coins? Any FIAT?
Today's FIAT have nothing to do with what currency, basically just a fancy IOU note, used to be. Currency WAS representative of human work. Directly. The oldest known form are lion teeth. You had to WORK to get them. It was direct proof of work. And it was important to keep that link so no arbitrary value is taken or added to it.
The next major leap of currency was IOU notes. Notes that banks respected between each other. One bank gives you some paper that says "trust me, bro, I'm worth 2 lion teeth". You go to another bank and can exchange that paper back to 2 lion teeth.
At one point in time, the US bamboozled the entire world and declared the US dollar as some form of new gold bar (what it was previously tied to, gold; proof of work). Tying all of FIAT to it. A value that can more easily be arbitrarily changed. And, since then, propaganda reigns supreme and people like you are born. Praising a nonsensical paper as some kind immovable artefact of mankind.
Language evolved. Law evolved. Currency didn't evolve. And, for some reason, you are fighting its evolution.
Don't vote.
The dollar is fundamentally a promise of the US government that they owe you a given amount of money. As long as the US government exists, a dollar is going to have a worth. You may debate the virtue of the US government all you want, but at the same time, there's nobody to give you any such guarantee for a Bitcoin, or an NFT. The moment the market settles for a new plaything, these binary numbers you praise will be entirely worthless. So when I say money, I'm referring to the trustworthiness of its issuer.
The only thing Crypto has brought us is a gambling system rich people use to get richer, and poor people use to loose money; lots of brainlessly burned electricity; a neat way to collect ransomware payouts; and a bunch of Ponzi schemes.
People like you like to feel smarter than anybody else, and I know you'll stay committed to cryptocurrency regardless of any reason. So I will drop out of this discussion here, as it's fruitless.
If you want to know the actual answer however. The answer is debt. Or more specifically assets and liabilities on various bank ledgers. Those banks organically set the value to monetary systems. This system would fall apart without a forced taxation system for a given currency.
The actual value is debt and taxation. I'm not complaining, who am I to criticize? But that is the answer.
One could conceive of such a crypto ledger system, and that would be a CBDC.
And if we consider a CBDC to still count as cryptocurrency, in my opinion we've just shifted the goal posts from a decentralised, anonymous, revolutionary, grass-roots form of digital currency to the technical underpinnings of the digitisation process of the global banking system.
Decentralization will incentivize hoarding. Why spend crypto when it might go to the moon tomorrow. Maybe that is a feature, maybe not.
I don't judge.
You wouldn't end up being "just another generation site with crypto", you'd become a magnet for everyone who's generating things they're scared of having their legal identity tied to.
No, I don't want to upload my passport and personal information to some site.
No, I don't want to use Paypal or credit cards
No, I'm not a criminal. I value my privacy, as many others do.
I had to stop supporting logged out users because 99% of the CSAM generated on the site came from people trying to skirt the basic login requirements.
Sorry but your privacy doesn't trump my not wanting to run a child porn site, and the overwhelming majority of the public manages to live through having to login in and pay for things with traceable money.
And before the conversation goes there: is it a perfect filter? No. But a 99% reduction (realistically near 100% reduction) in unwanted behavior is what I'd call very effective.
Because otherwise hackernews will be filled with spam. We don't want hackernews to be running a child porn site and this will reduce 99% of that risk.
Giving away your personal privacy isn't the only option and you know that.
It's in the title which is not immediately visible on mobile.
Unfortunately, I got an error signing up: "You have reached your limit of 500 users. If you need more users, please use a Production instance."
Btw, there is a stray semicolon on the signup page after the cardbox ;)
Signups are re-enabled, and I'm writing a script now to re-associate everyone's old chat history. Sorry for the launch day struggles!
Thanks for testing! :)
- Billy
We weren't really sure how to price it, so we're using the beta period for now to figure out what mix of models people are using and trying to figure out reasonable pricing based on that, and also ironing out various bugs and sharp edges. Then we'll start charging for it; personally I'd prefer to have it be usage-based pricing rather than the monthly subscriptions that ChatGPT and Claude use, so that you can treat it more like API access for those companies and don't have to worry about message caps.
It currently reads to me like you’re just having fun with it at the moment.
Sign up should be working again! Thanks for testing! :)
- Billy
It means that we can spin up a single model server and use it for multiple people, effectively splitting the cost. Whereas if you try to rent the GPUs yourself on something like Runpod, you'll end up paying much more since you're the only person using the model.
- Billy
My assumption is they're calling this ability multitenant.
"We weren't really sure how to price it, so we're using the beta period for now to figure out what mix of models people are using and trying to figure out reasonable pricing based on that, and also ironing out various bugs and sharp edges. Then we'll start charging for it; personally I'd prefer to have it be usage-based pricing rather than the monthly subscriptions that ChatGPT and Claude use, so that you can treat it more like API access for those companies and don't have to worry about message caps."
Open to feedback here! :)
- Billy
Application error: a server-side exception has occurred (see the server logs for more information).
Digest: 1138364229
I only get 503 Error response now when I try to load the page.