GPU-rich labs have won: What's left for the rest of us is distillation (inference.net)

The blog kept redirecting to the home page after a second, so here's an archive: https://archive.is/SE78v

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency.

But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot of researchers to try things that are really different. At least I hope so.

It seems pretty short sighted that the funding numbers for memristor startups (for example) are so low so far.

Anyway, assuming that within the next several years more radically different AI hardware and AI architecture paradigms pay off in efficiency gains, the current situation will change. Fully human level AI will be commoditized, and training will be well within the reach of small companies.

I think we should anticipate this given the strong level of need to increase efficiency dramatically, the number of existing research programs, the amount of investment in AI overall, and the history of computation that shows numerous dramatic paradigm shifts.

So anyway "the rest of us" I think should be banding together and making much larger bets on proving and scaling radical new AI hardware paradigms.

Not a fan of fear based marketing: "The whole world is too big and expensive for you to participate in, so use our service instead"

I'd rather approach these things from the PoV of: "We use distillation to solve your problems today"

The last sentence kind of says it all: "If you have 30k+/mo in model spend, we'd love to chat."

We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.

Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th.

GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.

If I'm understanding this correctly, we should see some great coding LLMs. Idk, could be as limited as a single stack eg laravel/nextjs ecosystem.

Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization.

Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday...

...and $3 trillion of Nvidia stock going down the drain by Friday.

Tropical Distillation?

50m per day is insane! Any link supporting that?

Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph:

> Impressively, open source models have been able to quickly catch up to big labs.

And then the beginning of the fourth:

> Open-source has been lagging behind proprietary models for years, but lately this gap has been widening.

Followed by a picture that is more or less inscrutable.

Have won what? The privilege of burning billions of dollars and not being profitable?

Until AGI is achieved no one's really won anything.