Yes, but still, what about users looking for NPU support. What happened to Microsoft's copilot+ PC campaign? (the NPU hardware, not the logo key and ad campaign)
AMD's XDNA2 NPU is best supported by ONNX, I think Qualcomm Hexagon is too. The best consumer unified architectures are Apple, AMD ai+ and Qualcomm Snapdragon X Elite, even intel has an NPU offering.
There are plenty of smaller less powerful open weights models if all you have to run it on is an NPU.
This is a much larger more powerful model, so running it locally will not be for everyone, but you can still run it via a hosting service such as Fireworks AI.
I guess that's the attraction - that they may be small enough (likely quantized) so that you can run them locally, so there is no privacy issues, and no-one can turn it off, etc, but technically "open weights" is just referring to models, of any size, where they give you the weights (basically give you the model) so you can run it yourself, instead of having to access it via paid API running on some computer that they control.
Apple research has also posted information on the topic.
> In this post, we show how to run the latest frontier-class model — Muse Spark Open, with nearly a trillion parameters — right from your desk, using the latest version of MLX, released today. On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second
Beginning on a single Mac Studio (up to 512GB of unified memory), then scaling out across multiple Macs with the RDMA over Thunderbolt driver, below we share an overview of the range of models you can now run on Mac Studio.
This is what stood out to me. They did the actual work of testing and documenting deployments on several platforms. It's a bit of upping the standard for open weights instead of just posting the blob and letting the community figure it out.
It makes sense from a provider standpoint. If you are going to give it away for free what is the end game? Helping build the world around your model such that it is a pathway that leads to greater adoption of your ecosystem is smart. Dev relations in a way.
Yes you need big compute to deploy this locally but $20-30K in hardware for specialty markets is nothing.
This isn't an accident. It is revealing a strategy.
11 comments
[ 2.8 ms ] story [ 106 ms ] threadThis is a much larger more powerful model, so running it locally will not be for everyone, but you can still run it via a hosting service such as Fireworks AI.
NPUs are not inherently useless, but definitely so the way they are forced onto users machines.
> In this post, we show how to run the latest frontier-class model — Muse Spark Open, with nearly a trillion parameters — right from your desk, using the latest version of MLX, released today. On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second
Beginning on a single Mac Studio (up to 512GB of unified memory), then scaling out across multiple Macs with the RDMA over Thunderbolt driver, below we share an overview of the range of models you can now run on Mac Studio.
https://machinelearning.apple.com/research/mlx-distributed-i...
It makes sense from a provider standpoint. If you are going to give it away for free what is the end game? Helping build the world around your model such that it is a pathway that leads to greater adoption of your ecosystem is smart. Dev relations in a way.
Yes you need big compute to deploy this locally but $20-30K in hardware for specialty markets is nothing.
This isn't an accident. It is revealing a strategy.