Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.
Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
Reading an article takes more time than commenting, which takes more time than voting. So the typical life cycle of a HN post is that people vote on it without reading the article, comment without reading the article, vote on the comments without reading the article, and respond to comments without reading the article, until after some time people who've actually read the article show up in the comments with some clarifications, saving everyone else the bother of having to read it themselves.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
Yeah, I was disappointed reading the title of this post because the work it is referencing was very exciting. But after reading this post, it's clear that the author is just pulling an "uhm, ackchyually" and responding to an abstract audience of laymen readers who don't know (or care) about any of the details of these results.
Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.
You don’t think the authors sort of misrepresented these findings in a way such that they sort of hooked readers in with enticing promises while glossing over major details that demonstrate they are over selling their research….? In this day and age the second I detect that kind of shit I have to discredit the paper and or take it with nothing more than a grain of salt or two. They should be more upfront and clear with what they did from the first paragraph. Instead they made you dive into details to read their stuff only to find out their headline wasn’t exactly telling the full story here.
The title of their research is "LLMs can natively read & write neural network weights". That's accurate and not sensational at all. People made posts on HN, etc., hamming up these results, but it seems like the actual authors of this research presented it professionally.
>They should be more upfront and clear with what they did from the first paragraph.
LLMs are extremely hyped up right now. It's like when everyone was crypto-crazy, and threads were full of people who had no idea what was going on but were very excited.
As someone who has been working in ML for 30+ years, and has designed and trained too many NNs to keep count, this is the first time I've seen anyone twiddling the weights directly. Usually you play with the training set, exemplar weights, model's hyperparameters, etc. I've never seen a trained network being modified by hand.
it is manual, just automated via running inferences on a dataset of assumed refusals, and is a small rank-1 weight edit that doesn't require fine-tuning or training. Perf hit is much smaller than baking in these refusals in the first place.
There’s some Dunning-Kruger effect going on. Many are looking to more knowledgeable commentators in hackernews for a digestible eli5 takeaway, myself included as the original article flew over my head.
Unfortunately it’s hard to distinguish who to trust in threads as many commentators sound overly confident and knowledgeable. They think they understand the article and its nuances, but they actually don’t and end up perpetuating false info.
I strongly recommend avoiding HN for intelligent takes on AI internals. This place is full of people who think they can understand these models by vague analogies to the brain and their own pet (and almost-always underdeveloped and philosophically ignorant) theory of mind, and also that you can have a meaningful understanding of these models without at least knowing some very basic linear algebra.
I would recommend going on Twitter (or a Nitter clone) and looking at e.g. Neel Nanda, Nick Camarata, roon, Jon Stokes, or other researchers that are directly involved in this stuff. HN when AI is involved is full of the most incompetent and embarrassing armchair speculators saying things so incoherent they aren't even wrong https://en.wikipedia.org/wiki/Not_even_wrong.
> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.
I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.
If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
I dunno, I'm still impressed by the "Sparks of Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could do understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. I'm sure humans can build tooling to also modify weights, but the author here agrees that it's tough. The examples where Astra was modifying raw weights are still novel.
Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.
Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced
A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.
Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).
Yes but my point is that I never realized that the AI would be writing tools to update the weights directly, rather than improve the existing training process.
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.
They are also biased to see from what institution you come from..
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.
Considering I have a PhD in CS focused in NLP and have been a Senior Area Chair for ACL, yes I do know how academia works. And, no this isn't science working; it's the same plague infecting most of academic blogging and social media. People are vying for attention in whatever way possible, often at the expense of others.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
All I can say is I very much disagree, and if I had authored the post and it had received the reaction from people that it has I would be anxious to issue corrections. I haven't reviewed for conferences for several years now, but I wouldn't have thought this should pass review.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.
If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
> Because this is very much a part of what ML engineers do.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
I think you misunderstand. Yet the model is doing something impressive still by writing code to fix model weights on problems. But that's a whole different ballgame than just reading weights and knowing what to change. That would mean that they have an internal representation of how the model weights map to information or a specific task, which if true, throws all the interpretability research out of the window
To me, the original article is more about what LLMs can *do* in weight space, I don't really see the part where it would tie to an internal representation.
163 comments
[ 8.2 ms ] story [ 131 ms ] threadSure if you only read the title and brief at the top you’d walk away without the full picture though.
This is a better response imo: https://news.ycombinator.com/item?id=50344861
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.
>They should be more upfront and clear with what they did from the first paragraph.
Please read the first paragraph and tell me what isn't upfront or clear about it: https://rikiyat.github.io/blog/weight-space-programming
I'd say it's spelled out so clearly that even people who don't understand this stuff could confidentally understand what they're doing here.
Unfortunately it’s hard to distinguish who to trust in threads as many commentators sound overly confident and knowledgeable. They think they understand the article and its nuances, but they actually don’t and end up perpetuating false info.
I would recommend going on Twitter (or a Nitter clone) and looking at e.g. Neel Nanda, Nick Camarata, roon, Jon Stokes, or other researchers that are directly involved in this stuff. HN when AI is involved is full of the most incompetent and embarrassing armchair speculators saying things so incoherent they aren't even wrong https://en.wikipedia.org/wiki/Not_even_wrong.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
They are also biased to see from what institution you come from..
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
I don't know why because Anthropic has had a lot of blog posts about internal work on it
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...
Trying to find every single angle and option to break something is how we verify the truth of something.
If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.
This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.
So I do agree it is misleading and trying to present work as groundbreaking.
The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.
There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!