What hardware are you using to run local OS models?

I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.

[flagged]

I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )

Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).

I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.

I run a cluster of two Dell Pro Max GB10's 128GB unified mem per node 200G NIC ConnectX

Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.

I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings. I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.

I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.

Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.

MY SYSTEM: ASUS ROG MAXIMUS XI HERO Motherboard Intel® Core™ i9-9900K × 16 64GB DDR5 RAM RTX 3090 with 24GB VRAM

I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.

I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)

A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think

Upgraded from a 3060 12G to a 4070 12G.

r9700s - running qwen 3.8 flash next with ~1m token kv cache

Qwen Next 3.8 @ 3 bit IQ on x2 Quadro 6000 (sm_75) on a Dell T5820 (W-2140B Xeon, 128GiB DDR3). Strata makes the most of the hardware.

heavily customised agentic framework: https://github.com/connollydavid/host gets good work out of the weights

I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.

Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.

I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).

I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.

Generic setup doc here: (not fully updated with GUFO): https://github.com/Selicos/AI_Tools_Public/blob/main/docs/Fe...

Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.

I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.

Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.

I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.

IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.

Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.

My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.

Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.

I feel very poor with my M2 MBP with only 16G of RAM

I have a pair of ASUS GX10 systems [0] (DGX Sparks in a different enclosure) in a cluster running DeepSeek-V4-Flash-0731 [1] with tensor parallelism. I managed to snag them early this year when prices were merely stratospheric, not mesospheric.

Even at half of today's retail price they're not remotely cost-effective compared to cloud models for my single-developer workflow, and idle power draw is quite high with the ConnectX-7 NICs powered on for clustering. (Mine idle around 45-50W each. Powering off the NICs cuts that roughly in half, but they're still very power-hungry compared to an Apple Silicon Mac or even a Framework Desktop.)

I learned a lot building the cluster, which was my main objective. I could probably sell them for a nice profit, but I'm still enjoying using them.

[0] https://www.asus.com/networking-iot-servers/desktop-ai-super... [1] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731

Local clusters built around old GPUs. I have had cluster of P40 GPUs, MI50s gpu, and now I have a cluster of cmp170hx. I'm always looking for the next cheap GPU dump and then I get as many as I can before it get's popular.

I have a Mac Studio M1 Ultra with 128GB of RAM, an M5 Pro MBP with 64GB of RAM, and until recently an M1 Pro MBP with 32GB of RAM. I also have two Mac Minis with 16GB of RAM, but they're mainly used for embedding models because they're so small.

I'm doing a bunch of batch work for an experiment with them right now, but my daily driver is a Gemini Pro subscription.

[flagged]

Got a refurbished M1 Max 64GB for less than USD 1,000 some years ago; runs medium models at decent speed. Large ones are out of the question.

[flagged]

I'm planning to run something akin to https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... on my Mac Studio M3 Ultra with 96 GB RAM. It will be the main orchestrator of my entire system.

The plan is to use it with Open WebUI, PostgreSQL with pgvector and Apache AGE, and my own database schema, and then have it be my complete second brain, running completely privately on my own hardware and always being available over Tailscale.

Since I still have 20-30 GB RAM to spare after loading that main orchestrator I will also have a coding worker that will do most of the grunt work, coordinated and commissioned by the orchestrator. For now I have readied https://huggingface.co/pyros-vault/Ornith-1.5-35B-A3B-BigBan... (a mix of Qwen 3.5 35B and BigBang) for this task.

For now I still use Codex and ChatGPT Pro 200 to do all my work, but the plan is to move over to local LLM work as soon as I have finished everything I realistically need frontier models for. Which is going to be for a little longer.

my 'local llm on a budget' has cost me just about $2,400 in total, spread over a bout 3 years of accumulating stuff.

I got my first 7900xt 20gb on a good sale shortly after they came out in an open box deal on ebay for $590.

After another 7900xt 20gb popped up on facebook marketplace recently for $420, I sniped that and then got a microcenter deal on an asus proart board, 32gb of ddr5, and a 9950x for $1050.

A couple more deals for used cases, an older 1tb nvme ssd I had lying around, and a nice PSU finished the build.

I run Qwen 3.8 27b on linux and have tuned the llamma.ccp / rocm setup substantially with the help of codex and claude. I get ~80-110 tok/s depending on all kinds of factors and full 256k context windows with a 6-bit quant.

It's surprisingly capable and I love that I can run it indefinitely, tuning as it goes to behave more like a frontier model.