great, I've been reading series of papers related to open source/open weight LLM Backdoors.
It's 100% possible, curious when will be the first incident.
Having something built by entity with HQ in the wester world is always a bit safer. That could be mistral, but they decided to be a consultancy company, not an ai lab.
everyone is doing backdoors, except eu, they just receive bribes from those who install backdoors.
the only difference is: the backdoor is made to kill people like you or not. I'd rather have a backdoor that is installed to kill someone on the different continent, e.g Iran leadrship, not in my country/city.
This post could be framed with golden filigree, titled "The Most HN Comment Ever Written" and perhaps displayed near other artworks like a Juicero, Flock camera or Sam Altman's disinterred belongings.
Orange Site will have a properly interesting history museum, at the rate we collect curiosities like this.
Nemotron, Inkling, Gemma, Trinity Large were all released/updated in the last 6 months, and there are countless other smaller models out there. US open weight is very far from "dead".
How does this compare to Muse Spark 1.3 contribute? I've used Spark 1.3 a lot lately , since its currently free to use via OpenCode. I haven't had the best experience in the past with models from Meta. But Spark 1.3 is has been surprisingly great. Especially for tool calling.
It's really nice that all AI pages are using canvas animations. A few days ago, Gemma 4's animation wasn't that impressive. Same with Muse. So far, the best animations I've seen are from OpenAI and Anthropic.
Ehm.. I have a model that runs on my older gaming desktop that scores as well (or better) as this model does on DeepSWE. This model does better than Flash Next in many areas and I am sure is much stronger overall, but hmm.
re: "Apple’s unified memory architecture lets the M5 Ultra's integrated GPU use up to 512 GB per Mac Studio, so two machines can hold an entire model. And the chip’s 1.2 TB/s of memory bandwidth is fast enough to read the small slice each token needs. The result is two Mac Studios behave like one supercomputer with a terabyte of unified memory."
At the current market price of Apple Mac Studio Ultra, 30-core CPU + 64-core GPU w/ 256GB RAM is already reaching at around $10k. Keep in mind that computers are appreciating assets recently.
The 512GB RAM option is not even for sales yet (they said late October availability). Based on typical Apple pricing, it's likely to be around $14k apiece. So we are looking at least $28k - $30k for the setup Muse team is advertising. That's essentially the Nvidia H100 GPU w/ 96GB VRAM/HBM3 at up to 3.94 TB/second memory bandwidth, 3x+ than Mac Studio's Unified Memory speed.
Also, how did Meta get access to Apple's unreleased M5 Ultra w/512GB RAM Mac Studio? Or is that simply a marketing piece that they wrote purely on specs?
> Also, how did Meta get access to Apple's unreleased M5 Ultra w/512GB RAM Mac Studio? Or is that simply a marketing piece that they wrote purely on specs?
It should be well known that Apple provides early access to developers under strict conditions. Otherwise, how do you think they announce new hardware not for sale, and show demos of third party apps taking advantage of such hardware?
The 512GB is obviously geared towards local LLM crowds. I'm just speculating, but I would be surprised if there's not at least a few devices with the 'ecosystem players' like LM Studio, Unsloth, etc. Of course, they'd never confirm nor deny :)
Well, this is much better than nothing -- we should be grateful that whatever the market conditions and corporate incentives, a major lab is releasing a capable open-weight model. But still very far behind DeepSeek and GLM. Even on consumer hardware, Qwen 3.8 27B is going to compare favorably to this.
True, but the conditions are quite relaxed, so it can even be used commercially without worries by small businesses, while it is unconditionally free for non-commercial uses.
Other big open-weights LLMs have similar licensing terms and I believe that they are quite fair, i.e. at the point when someone uses such an LLM to obtain tens of million $ of revenue it becomes fair to demand a share of that for the creator of the model.
Not at an appreciable token rate.
I wonder if these large open weight models are considered safer to release for that reason?
I'm thinking no because any small to medium size criminal organization or state funded group will be able to cobble together the resources to achieve some kind of runtime.
Even at 10 tok/sec, making bioweapons at 10 tok/sec is concerning.
You do realize that a high school Chemistry course actually contains all the details you need on both bioweapons used in WW2, right? Plus several others you could use. Mustard gas and Zyklon-B, including preparation instructions, but of course, you'll have to puzzle it together. That these substances can be used as bioweapons is heavily deemphasized, but you just can't reasonably avoid mentioning and showing them.
Also these books contain maybe not the solution to making thermobaric explosions, but certainly the principle and show you exactly what problem you'd have to solve to make those weapons.
Of course, in both cases, the real challenge is making them and deploying them without killing yourself.
We are not safe from bioweapons because nobody knows how to make them. That's not it.
It seems like you're talking about chemical weapons.
Which is fair, they are very easy to make without the assistance of an LLM, and I don't hear about that kind of thing in the news.
I was talking about biological weapons like gain of function and that kind of thing.
The implied question was 'Should companies like Meta publish open weight models when they could theoretically be subverted by malicious users?'. So far as I'm aware there are few people worrying about this kind of thing. Bill Gates for one. But apparently it's off topic and/or not interesting, or not welcomed as a discussion point.
You were worried about someone using LLMs to make weapons. So I went for the ones that are more realistic. Bioweapons are far harder to make than chemical weapons. The science is harder, the actual synthesis can only happen at a few biomed companies (which is where I'd put my checks if I were you), even when you have the DNA synthesis is very hard, in the "just don't bother before your PhD hard" realm, and they're even harder to handle without killing yourself. Bioweapons are not realistic.
I've always thought Bill Gates was pretty smart, smarter than me, and has dedicated a lot of his time to researching things like pandemics. More time than I have dedicated to such research.
Seems like Meta's release of Muse Spark Open does not move the needle on things like this, since GLM-5.3 was already released back in August.
I do not know about you, but I will probably also run this model, albeit very slowly.
Even if for your main LLM you want a speed high enough for interactive use and fast iterations, it is still useful to also run a much bigger LLM from time to time, even if that requires many hours for completing a task, due to the slow inference speed when the weights are read from SSDs.
There are tasks where the quality of the result is more important than the time needed to produce it.
I tried Muse Spark 1.3 for a while because it's so cheap, but it's definitely in a lower league than Astra, Opus 5.5 and Argon. Still produces slop if you try and vibe code with it, whereas the more capable models write good code.
Tech people really are drug addicts. Rant about Zuck and Meta for 364 days a year and then jump over each other to be first in line the moment he shows something shiny.
This is great progress for "open-weight" models. Quotes due to the restrictions, but still progress none-the-less. Looking forward to seeing any quants that all mere mortals to run this at home without early access Apple hardware.
293 comments
[ 6.3 ms ] story [ 207 ms ] threadIt's 100% possible, curious when will be the first incident.
Having something built by entity with HQ in the wester world is always a bit safer. That could be mistral, but they decided to be a consultancy company, not an ai lab.
the only difference is: the backdoor is made to kill people like you or not. I'd rather have a backdoor that is installed to kill someone on the different continent, e.g Iran leadrship, not in my country/city.
Orange Site will have a properly interesting history museum, at the rate we collect curiosities like this.
Speaks nicely, great at driving linux. Not frontier-level coding, but comparable to luna, deepseek. It was amazing for the price.
But weaker than Muse Spark 1.3: https://i.imgur.com/8ST1rRY.png
At the current market price of Apple Mac Studio Ultra, 30-core CPU + 64-core GPU w/ 256GB RAM is already reaching at around $10k. Keep in mind that computers are appreciating assets recently.
The 512GB RAM option is not even for sales yet (they said late October availability). Based on typical Apple pricing, it's likely to be around $14k apiece. So we are looking at least $28k - $30k for the setup Muse team is advertising. That's essentially the Nvidia H100 GPU w/ 96GB VRAM/HBM3 at up to 3.94 TB/second memory bandwidth, 3x+ than Mac Studio's Unified Memory speed.
Also, how did Meta get access to Apple's unreleased M5 Ultra w/512GB RAM Mac Studio? Or is that simply a marketing piece that they wrote purely on specs?
It should be well known that Apple provides early access to developers under strict conditions. Otherwise, how do you think they announce new hardware not for sale, and show demos of third party apps taking advantage of such hardware?
The 512GB is obviously geared towards local LLM crowds. I'm just speculating, but I would be surprised if there's not at least a few devices with the 'ecosystem players' like LM Studio, Unsloth, etc. Of course, they'd never confirm nor deny :)
Uses Exo - https://exolabs.net
You can do it with other software, that's just one of the quick-and-easy setups if you've got a spare 40-60k for studios sitting around
So long as you can choose between running AI locally or a new Corolla
You are correct but at my age my brain is never going to stop being surprised by this statement.
Other big open-weights LLMs have similar licensing terms and I believe that they are quite fair, i.e. at the point when someone uses such an LLM to obtain tens of million $ of revenue it becomes fair to demand a share of that for the creator of the model.
You trust anything he says?
Also these books contain maybe not the solution to making thermobaric explosions, but certainly the principle and show you exactly what problem you'd have to solve to make those weapons.
Of course, in both cases, the real challenge is making them and deploying them without killing yourself.
We are not safe from bioweapons because nobody knows how to make them. That's not it.
Which is fair, they are very easy to make without the assistance of an LLM, and I don't hear about that kind of thing in the news.
I was talking about biological weapons like gain of function and that kind of thing.
The implied question was 'Should companies like Meta publish open weight models when they could theoretically be subverted by malicious users?'. So far as I'm aware there are few people worrying about this kind of thing. Bill Gates for one. But apparently it's off topic and/or not interesting, or not welcomed as a discussion point.
https://youtu.be/tj1dFIUAs7A?si=a4KYg6QEaxALHnWX&t=207
I've always thought Bill Gates was pretty smart, smarter than me, and has dedicated a lot of his time to researching things like pandemics. More time than I have dedicated to such research.
Seems like Meta's release of Muse Spark Open does not move the needle on things like this, since GLM-5.3 was already released back in August.
Ouch. I wonder how it compares to Qwen 3.8 running with 4-bit quant on 10 times less VRAM.
Even if for your main LLM you want a speed high enough for interactive use and fast iterations, it is still useful to also run a much bigger LLM from time to time, even if that requires many hours for completing a task, due to the slow inference speed when the weights are read from SSDs.
There are tasks where the quality of the result is more important than the time needed to produce it.
Do not run, do not support, do not use.
PS: I might need to do a doom benchmark, that’d be fun https://playground.jeffyclassify.com/#doom
Is that combination of tokens still relevant or did it leave the news cycle by now?
I don't trust any benchmark numbers from Meta anymore.