HN Explorer

AskHN: What's a real world production use case for Ollama?

11 points by eclectic29 ↗ HN
Ollama is enjoying a LOT of hype, but I'm struggling to find a real world production use case for it. Where might I really want to use this? It's a wrapper around llama.cpp and makes it easier to download LLMs. Where might I want to download models in production like this? In production I would rather deploy thoroughly tested models. Plus, the model offloading capability on the fly is really not meant for production as it hinders performance. Thoughts?

6 comments

[ 3.4 ms ] story [ 27.8 ms ] thread
Tomte
Not everything must be about commercial use. ollama is a great way to get from "I have no idea what this LLM stuff is" to "cool, I‘m already talking to one?".
runjake
It’s not for production. Ollama’s documentation explicitly warns you not to use the API in production. It’s for messing around and experimenting with LLMs and different models.

If you don’t see the use of it, I recommend you use something else that suits your tastes more.

eclectic29
Thanks. That’s fair. Just shocked by the extreme hype.
eclectic29
Btw, where did you see that line of text in the docs? I didn't find it.
speedgoose
We use it in our research group to run the largest LLMs quickly on a descent server. However, the people with the latest MacBooks don’t bother and prefer to use their laptops as they are fast enough.

We also throw money at Microsoft and OpenAI when we evaluate ChatGPT. But open LLMs are cheaper to play with if you already have the hardware. And more aligned to our research values.

yen223
Didn't know Ollama was used in production. I use Ollama + the Ollama Raycast plugin as it's probably the quickest way to run an LLM locally