HN Explorer

The $2k Laptop That Replaced My $200/Month AI Subscription

8 points by Raywob ↗ HN
Cloud AI pricing is per-token. The more useful your pipeline, the more it costs. I built a dual-model orchestration pattern that routes 80% of work to a free local model (Qwen3 8B on Ollama, GPU-accelerated) and only sends the synthesis/judgment stage to a cloud API.

Cost for a 50-item research pipeline: $0.15-0.40 vs $8-15 all-cloud. Same output quality where it matters.

Stack: RTX 5080 laptop, Ollama in Docker with GPU passthrough, PostgreSQL, Redis, Claude API for the final 20%.

The pattern: scan locally → score locally → deduplicate locally → synthesize via cloud. Four stages, three are free.

Gotchas I hit: Qwen3's thinking tokens through /api/generate (use /api/chat instead), Docker binding to IPv4 only while Windows resolves localhost to IPv6, and GPU memory ceilings on consumer cards.

Happy to share architecture details in comments.

3 comments

[ 2.7 ms ] story [ 18.3 ms ] thread
solomatov
How is quality for what Qwen 8B provides compares to proprietary models? Is it good enough for your use case?
roosgit
Have you tried other local models?

The 14B Q4_K_M needs 9GB, but Q3_K_M is 7.3GB. But you also need some room for context. Still, maybe using `--override-tensor` in llama.cpp would get you a 50% improvement over "naively" offloading layers to the GPU. Or possibly GPT-OSS-20B. It's 12.1GB in MXFP4, but it’s a MOE model so only a part of it would need to be on the GPU. On my dedicated 12GB 3060 it runs at 85 t/s, with a smallish context. I've also read on Reddit some claims that Qwen3 4B 2507 might be better than 8B, because Qwen never released a "2507" update for 8B.

frigosk
The hybrid local/cloud orchestration pattern is a smart way to manage costs, especially for pipelines with distinct stages. We built a similar routing logic into https://simplio.dev to handle the fallback and provider switching automatically, which saves a lot of custom pipeline code.