GPT-OSS-120B runs on just 8GB VRAM & 64GB+ system RAM (old.reddit.com)

[flagged]

I have a 5950x with 128 gb ram and a 12 gb 3060 gpu. The speed of generating tokens is excellent, the killer is that when the context grows even a little processing of it is super slow. Hopefully someone smart will optimize this, but as it is now I keep using other models like qwen, mistral and gemma.

If you run these on your own hardware can you take the guard-rails off (ie "I'm afraid I can't assist with that"), or are they baked into the model?

LLM noob here. Would this optimization work with any MoE model or is it specific for this one?

[flagged]

Don’t have enough ram for this model, however the smaller 20B model runs nice and fast on my MacBook and is reasonably good for my use-cases. Pity that function calling is still broken with llama.cpp

I wonder if the mlx optimized would run on 64gb mac

I find it funny that people say "only" for a setup of 64GB RAM and 8GB VRAM. That's a LOT. I'd have to spend thousands to get that setup.

I wonder if GPT 5 is using a similar architecture, leveraging all of their data center deployments much more efficiently, prompting OpenAI to want to deprecate the other models so quickly

Is there a way to tune OpenWebUI or some other non-CLI interface to support this configuration? I have a rig with this exact spec, but I suspect the 20B model would be more successful.

I'm a little confused how these models run/fit onto VRAM. I have 32gb system RAM and 16gb VRAM. I can fit the 20b model all within vram, but then I can't increase the context window size past 8k tokens or so. Trying to max the context size leads to running out of VRAM. Can't it use my system ram as backup though?

Yet I see other people with less resources like 10GB of vram and 32gb system ram fitting the 120b model onto their hardware.

Perhaps its because ROCm isn't really supported by ollama for RDN4 architecture yet? I believe I'm using vulkan to currently run and it seems to use my CPU more than my GPU at the moment. Maybe I should just ask it all this.

I'm not complaining too much because it's still amazing I can run these models. I just like pushing the hardware to its limit.

Has anyone got it to run on Macbook Air M4 (the 20B version mind you) and/or an RTX 3060?