HN Explorer

Ask HN: Is there any good open source model with reliable agentic capabilities?

5 points by baalimago ↗ HN
I don't want to send my data to third party vendors all the time. But from my experience, the LLMs needs to be quite beefy in order to understand tool-calling, especially at longer contexts (200k+).

Before I dive headlong into investigating this and spend money on a project doomed to fail, do anyone have experience with a local model which can handle this sort of workload? I intend to run it on decent gaming CPU with 64-128GB ram.

2 comments

[ 3.7 ms ] story [ 19.0 ms ] thread
throwaway2027
CPU? Good luck.
baalimago
WDYM? I don't want to train a model, only use inference. From what I know it must be much cheaper to buy "normal" ram + a decent CPU vs a GPU with similar amounts of vram.

The bottleneck of the inference is fitting a good enough model into memory. A 80B param model 8bit fp quantization equates to roughly ~90GB ram. So 2x64GB DDR4 sticks is probably the most price efficient solution. The questions is: Is there any model which is capable enough to consistently deal with an agentic workload?