Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.
I also did a couple of experiments with pruning LLMs[1] using genetic algorithms and you can just keep removing a surprising amount of layers in big models before they start to have a stroke.
I think it's kinda ironic (in a more meta Alanis Morissette way) for such an article that has interesting content to default to have an LLM write it. Please, I request authors - you're better than this, and many people actually want to hear you, not an LLM.
For example, what really is the meaning of this sentence?
> These aren't just storage slots for data, they're the learned connections between artificial neurons, each one encoding a tiny fragment of knowledge about language, reasoning, and the patterns hidden in human communication.
I thought parameters were associated with connections, is the author implying that they also store data? Is the connection itself the stored data? Is there non-connective information that stores data? Or non-data-storage things that have connectivity aspects?
I spent a solid amount of time trying to understand what was being told, but thanks to what I would call a false/unnecessary "not just x but y" troupe, I unfortunately lost the plot.
IMO, a human who's a good writer would have a sentence that's clearer to understand, while non advanced writers (including me, almost certainly) would simply degrade gracefully to simpler sentence structure.
For people who want to dig deeper: The fancy ML term-of-art for the practice of cutting out a piece of a neural network and measuring the resulting effect on its performance is an ablation study.
Since around 2018, ablation has been an important tool to understand the structure and function of ML models, including LLMs. Searching for this term in papers about your favorite LLMs is a good way to learn more.
> The redundancy we observe in language models might also explain why these systems can generalize so effectively to tasks they were never explicitly trained on
What year was this written? 2023 and reposted in 2025? Or is the author unaware that the generalization promises of early GPT have failed to materialize and that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation, which has driven the progress of all models over the past few years.
6 comments
[ 2.9 ms ] story [ 20.4 ms ] thread[1]https://snats.xyz/pages/articles/pruningg.html
I think it's kinda ironic (in a more meta Alanis Morissette way) for such an article that has interesting content to default to have an LLM write it. Please, I request authors - you're better than this, and many people actually want to hear you, not an LLM.
For example, what really is the meaning of this sentence?
> These aren't just storage slots for data, they're the learned connections between artificial neurons, each one encoding a tiny fragment of knowledge about language, reasoning, and the patterns hidden in human communication.
I thought parameters were associated with connections, is the author implying that they also store data? Is the connection itself the stored data? Is there non-connective information that stores data? Or non-data-storage things that have connectivity aspects?
I spent a solid amount of time trying to understand what was being told, but thanks to what I would call a false/unnecessary "not just x but y" troupe, I unfortunately lost the plot.
IMO, a human who's a good writer would have a sentence that's clearer to understand, while non advanced writers (including me, almost certainly) would simply degrade gracefully to simpler sentence structure.
Since around 2018, ablation has been an important tool to understand the structure and function of ML models, including LLMs. Searching for this term in papers about your favorite LLMs is a good way to learn more.
What year was this written? 2023 and reposted in 2025? Or is the author unaware that the generalization promises of early GPT have failed to materialize and that all model makers actually have been training models explicitly on the most common tasks people use them for via synthetic data generation, which has driven the progress of all models over the past few years.