"How dare people I don't like give away software"
One does not simply "buy a few b200". They come in 8 packs minimally, at >5x that budget.
I don't know that "consumer hardware" is a useful distinction anymore, it's just "what's your budget and what's your speed requirement".
It can be more than "slightly", particularly if the model you're interested in (or will be interested in in 6 months) doesn't fit on the mac studio. You also need to account for eg storing 10TB of random checkpoints,…
The relevant comparison isn't one mac studio to one RTX 6000, it's a 24 channel DDR5 system, which also has ~1.2TB/s of memory bandwidth (or more when Xeon 6 compatible 8800mt/s memory becomes widely available), vastly…
It's unclear to me how bandwidth scales with multiple connections. Many-to-many does not seem ideal. Daisy chaining would be fine for straight pipeline work. There doesn't seem to be an equivalent of a ethernet switch…
It is. That's the mac mini. For local LLMs you would want the Mac Studio, which tops out at 1.2TB/s.
[flagged]
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
Most companies do not model themselves as "building on [AI model du jour]" yet. They model themselves as building products with those tools, which they consider as relatively substitutable.
"How dare people I don't like give away software"
One does not simply "buy a few b200". They come in 8 packs minimally, at >5x that budget.
I don't know that "consumer hardware" is a useful distinction anymore, it's just "what's your budget and what's your speed requirement".
It can be more than "slightly", particularly if the model you're interested in (or will be interested in in 6 months) doesn't fit on the mac studio. You also need to account for eg storing 10TB of random checkpoints,…
The relevant comparison isn't one mac studio to one RTX 6000, it's a 24 channel DDR5 system, which also has ~1.2TB/s of memory bandwidth (or more when Xeon 6 compatible 8800mt/s memory becomes widely available), vastly…
It's unclear to me how bandwidth scales with multiple connections. Many-to-many does not seem ideal. Daisy chaining would be fine for straight pipeline work. There doesn't seem to be an equivalent of a ethernet switch…
It is. That's the mac mini. For local LLMs you would want the Mac Studio, which tops out at 1.2TB/s.
[flagged]
[flagged]
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
Most companies do not model themselves as "building on [AI model du jour]" yet. They model themselves as building products with those tools, which they consider as relatively substitutable.