You might have a business idea there. I wouldn't mind a twinscan plushie for sitting on top of the workstation.
Yeah 48g, sub 200W seems like a sweet spot for a single card setup. Then you can stack as deep as you want to get the size of model you want for whatever you want to pay for the power bill.
Sure, all the slop code projects I produce get MIT licensed on public repos. It wasn't mine to begin with, so I wouldn't prevent anyone from using it.
Used 3090s have been getting expensive in some markets. Another option is dual 5060ti 16 gig. Mine are lower powered, single 8 pin power, so they max out around 180W. With that I'm getting 80t/s on the new qwen 3 30b…
About 768 gigs of ddr5 RAM in a dual socket server board with 12 channel memory and an extra 16 gig or better GPU for prompt processing. It's a few grand just to run this thing at 8-10 tokens/s
These articles are gold, thank you. I used your gemma one from a few weeks back to get gemma 3 performing properly. I know you guys are all GPU but do you do any testing on CPU/GPU mixes? I'd like to see the pp and t/s…
I run the KV cache at Q8 even on that model. Is it not working well for you?
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
You probably won't be running fp16 anything locally. We typically run Q5 or Q6 quants to maximize the size of the model and context length we can run with the VRAM we have available. The quality loss is negligable at Q6.
Just this pic: https://imgur.com/ip8GWIh
I don't have a video but here's a pic of the output: https://imgur.com/ip8GWIh
It's a 3b model so the creativity is pretty limited. What helped for me was prompting for specific stories in specific styles. I have a python script that randomizes the prompt and the writing style, including asking…
I have a mini PC with an n100 CPU connected to a small 7" monitor sitting on my desk, under the regular PC. I have llama 3b (q4) generating endless stories in different genres and styles. It's fun to glance over at it…
I had this same issue with incomplete answers on longer summarization tasks. If you ask it to "go on" it will produce a better completion, but I haven't seen this behaviour in any other model.
It's probably an evolution of the phi-1/1.5 "Textbooks are all you Need" training method: https://arxiv.org/abs/2309.05463
No joke, that would be an awesome LLM project name!
Top_p and top_k are pretty important concepts for LLMs same as temperature so P,K,C and F are underutilized
Hey emad, thanks for SD and this! What's the plan if Meta does Apache 2.0 for LLaMA? Just keep going and making the 30b and 65b or build different models?
Vicuna-13b (4bit) got the answer right, the first time as well.
I've noticed the same with my Asus TUF laptops. I've had 2 generations of the 15" models with Ryzen processors and adding a second stick seemed to "wake" them up in a noticeable way. They Ryzen memory controller really…
My dad builds houses in northern Ontario and heat pumps seem to be getting more popular on new builds. This is a place that regularly gets below -30C in the winter. The heat pump (usually a heat pump/AC combo unit) by…
Not a good fit for lifestyle. I'm rural, and I haul around ATVs and dirt bikes with the main destination being 300km away and even more rural. My pickup truck is a way better choice for that. I used to tow a utility…
I love this idea. I've been desperate enough to reuse gaskets in a few repairs, and suffer oil leaks down the line. Usually it was because of back ordered gaskets that would be weeks or months out. Riding season in…
Asus TUF series (A15). Inexpensive with really good hardware specs and upgradable ram, storage and wifi cards. I usually go AMD on them, and midrange video cards. They're shockingly good as desktop replacements but they…
I believe this is mostly due to declining sales. My local dealer had good stock levels during the car shortages and the sales people were complaining about lots of Chargers, but no Grand Caravans. They could sell every…
You might have a business idea there. I wouldn't mind a twinscan plushie for sitting on top of the workstation.
Yeah 48g, sub 200W seems like a sweet spot for a single card setup. Then you can stack as deep as you want to get the size of model you want for whatever you want to pay for the power bill.
Sure, all the slop code projects I produce get MIT licensed on public repos. It wasn't mine to begin with, so I wouldn't prevent anyone from using it.
Used 3090s have been getting expensive in some markets. Another option is dual 5060ti 16 gig. Mine are lower powered, single 8 pin power, so they max out around 180W. With that I'm getting 80t/s on the new qwen 3 30b…
About 768 gigs of ddr5 RAM in a dual socket server board with 12 channel memory and an extra 16 gig or better GPU for prompt processing. It's a few grand just to run this thing at 8-10 tokens/s
These articles are gold, thank you. I used your gemma one from a few weeks back to get gemma 3 performing properly. I know you guys are all GPU but do you do any testing on CPU/GPU mixes? I'd like to see the pp and t/s…
I run the KV cache at Q8 even on that model. Is it not working well for you?
Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
You probably won't be running fp16 anything locally. We typically run Q5 or Q6 quants to maximize the size of the model and context length we can run with the VRAM we have available. The quality loss is negligable at Q6.
Just this pic: https://imgur.com/ip8GWIh
I don't have a video but here's a pic of the output: https://imgur.com/ip8GWIh
It's a 3b model so the creativity is pretty limited. What helped for me was prompting for specific stories in specific styles. I have a python script that randomizes the prompt and the writing style, including asking…
I have a mini PC with an n100 CPU connected to a small 7" monitor sitting on my desk, under the regular PC. I have llama 3b (q4) generating endless stories in different genres and styles. It's fun to glance over at it…
I had this same issue with incomplete answers on longer summarization tasks. If you ask it to "go on" it will produce a better completion, but I haven't seen this behaviour in any other model.
It's probably an evolution of the phi-1/1.5 "Textbooks are all you Need" training method: https://arxiv.org/abs/2309.05463
No joke, that would be an awesome LLM project name!
Top_p and top_k are pretty important concepts for LLMs same as temperature so P,K,C and F are underutilized
Hey emad, thanks for SD and this! What's the plan if Meta does Apache 2.0 for LLaMA? Just keep going and making the 30b and 65b or build different models?
Vicuna-13b (4bit) got the answer right, the first time as well.
I've noticed the same with my Asus TUF laptops. I've had 2 generations of the 15" models with Ryzen processors and adding a second stick seemed to "wake" them up in a noticeable way. They Ryzen memory controller really…
My dad builds houses in northern Ontario and heat pumps seem to be getting more popular on new builds. This is a place that regularly gets below -30C in the winter. The heat pump (usually a heat pump/AC combo unit) by…
Not a good fit for lifestyle. I'm rural, and I haul around ATVs and dirt bikes with the main destination being 300km away and even more rural. My pickup truck is a way better choice for that. I used to tow a utility…
I love this idea. I've been desperate enough to reuse gaskets in a few repairs, and suffer oil leaks down the line. Usually it was because of back ordered gaskets that would be weeks or months out. Riding season in…
Asus TUF series (A15). Inexpensive with really good hardware specs and upgradable ram, storage and wifi cards. I usually go AMD on them, and midrange video cards. They're shockingly good as desktop replacements but they…
I believe this is mostly due to declining sales. My local dealer had good stock levels during the car shortages and the sales people were complaining about lots of Chargers, but no Grand Caravans. They could sell every…