Ask HN: Could free/low cost LLMs be a momentary thing?

I think it’s very unwise to bet against the advancement of technology.

> Will we shift towards slower/worse LLMs running locally?

Bet on faster/better LLMs running locally and invest/brace yourself for recession accordingly.

Very clearly, all free LLM chatbots will need to be supported by ads. There is no other way to make them free in a large scale.

The rumour mill (justified, given cloud cost of running big open models with similar performance scores) says that these companies make money on inference, but lose it all on training.

So: when the money runs out and the bubble pops, we'll still get cheap existing models, what we lose is the race for new models.

We'd probably even keep free models: I forget where I saw it, but back in the early days someone noticed that models were so cheap that you could generate a decent sized blog post about any topic for about the same as the expected revenue from putting a few adverts on it and having it viewed *exactly once*.

That said, when (/if) these businesses stop chasing new models, it can make sense to burn the weights of the best at that date into a fixed (and analog, given how well they work with only a few bits of precision) circuit, making them more efficient. Not my field, so I'm not sure exactly how much more efficient analog can be; one or two orders of magnitude from what I've heard, but don't hold me to that, not my field.

Is it possible that at a point in the near future, when everyone is dependent on them, they can then remove all free tiers and pump up the prices?

But that would lead to another competition on prices again.

Yes. That’s my opinion. I think Apple will leverage their shared RAM and M architectures to sell their computers as local-LLM ready.

They for sure are testing LLMs and checking the performance of local models. Once they reach a performance and quality enough for some tasks they will announce Apple AI or some variation of the name.

All of this is speculation, but I think is obvious the right way.

Yes. I believe that. It’s just matter of time they start charging us quite a bit. And we won’t complain as we are so used to using by then

The best of the best will remain around the same price and potentially get more expensive as compute demand increases. However the intelligence/dollar ratio will increase over time due to a few factors (this is common in most tech)

1. Quantization: bigger models can be compressed and retain most of their quality, but run with a far smaller compute footprint

2. Moore's law: chips still are scaling so you can run more compute cheaper as it improves

3. Open source competition: if models from Anthropic or OpenAI get too expensive people will opt to use open source Chinese models, which would reduce demand and thus reduce prices down to a more reasonable equilibrium

Yes, I believe that will be the case. IBM already made the bet with granite models.

Personally, I've found that with guardrails, local 8-14B models can match frontier models on agentic tasks. The key is simple tasks with volume. For very complex things, the big models win. But a simple HR agent auditing 30,000 employee records to make sure all your info is filled out correctly, one at a time? You don't need frontier size for that.

I've been playing with local LLMs on a 4 year old Mac Studio and have been quite impressed. I keep remembering that the LLM I have available to me today is the worst it will ever be. It will just keep getting better. I'm super interested in and curious about the path forward for local inference.