Ask HN: Which open source LLM works best locally, and on what hardware?

1 points by donkan168 ↗ HN

10 comments

[ 0.27 ms ] story [ 36.2 ms ] thread
You can try: https://runitlocal.io/

Saw it on here a few days ago, you put in your GPUs and it helps give you an idea of what you can run. Although, the specific model weight is dramatically altered by use case. Do you want code creation? Creative writing? Assistant workloads? I would suggest doing some Googles/AI searches on “[Model returned by RunItLocal] for [use case]”.

thanks, that site is useful, do you run a model on your own machine? if yes, what hardware and model are you using, and is it good enough for coding?
I don’t run my own model, unfortunately I didn’t jump into getting a better GPU when I had the chance so I’m waiting out the market. In my specific use-case I use DeepSeek for programming since it’s the best bang for buck.
if you have a GTX 1060 6GB this might help you: "https://github.com/jpsanders/pascal-llm-stack"
thank you! I'm researching some information to understand the best setup (cost vs. performance and reasoning) to build a desktop computer for running an LLM, specifically for general tasks and reasoning in coding, that make sense?
Whatever model you decide on, look for an uncensored version. They are more likely to do "unsafe" things, like tell you what happened in Tiananmen Square in 1989, but they'll also accept what you ask or say at face value, without being argumentative or deferring to professionals. They generally run better and don't get stuck in loops as easily, too.
Depends on your gig. If you have a Mac with 48Gb or more, you can use qwen 27b model. It’s pretty good