1. Drops compute required for training by ~20%. This approach wont just help the ever escalating model sizes larger companies are pushing for, it means things like autoresearch can iterate on new model architectures faster.
2. WAY lower bandwidth requirements for inference. Means with approaches like this it should run on consumer hardware far better. It apparently requires 1/6th the memory bandwidth of a traditional approach for better results.
This is a big improvement if it can be generalized. They're claiming it's a drop in replacement, so it seems like it can as well.
> Abstract: Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. [...]
> Full AttnRes is straightforward but requires O(Ld) memory at scale. Block AttnRes partitions layers into N blocks, accumulates within each block via standard residuals, and applies attention only over block-level representations. With ~8 blocks, it recovers most of Full AttnRes's gains while serving as a practical drop-in replacement with marginal overhead.
Sadly, not the US either. It entirely cultural. Starting around Victorian times wealthy people took great pride in children who didn't work, fast forward to the industrial revolution a symbol of poverty was the child being the main breadwinner. Combine both of those class signalling methods and you get children who are locked out of work, but this manifests as forced busywork, electives, sports & music
In China since this class doesn't exist and forced busywork for class signaling is largely driven underground due to government policies driven young people just work on whatever the government policy says, which right now is tech. Hence you see smart young people getting into tech. While in the US they do Finance (if they built up the right portfolio to be allowed in the elite jobs club), speedrunning (if they didn't). Ironically the current US look a lot like imperial China, while China looks like the US of the late 1800s, but fully vertically integrated
Long, long ago, I remember the first toy I ever got that was made in chiina. It was a wooden cube puzzle. various interlocking differently shaped pieces that when assembled formed a cube. It was so different to all the other toys made in america by hasbro, mattel, tonka, etc. Back then I felt like I was holding a toy made by the ancient Greeks, a puzzle to teach geometry, analysis, pattern recognition. So abstract, so removed from daily life, it transported me into a different world. Like chess, it was an engaging abstraction. But unlike chess, it was not about conflict but rather interrelating pieces to make a greater whole.
So this is what really unsettles me. Not that China graduates more engineers every year than we have entirely employed in the US, but rather, that these individuals are not about delegating work, but actually doing it. Whereas the western credo is to get someone else to do the work (or in the words of PAtton, to get some one else to die for his country), I get the feeling that China will get robots and AI to do the work. I am reminded of the joke about Chinese factories having only 1 security guard and 1 dog. The guard is there to feed the dog.
I don’t want to discount the hard work that went into this, but I think it is always important to recognize this is mostly not a representation of competence, but of means and access. I have never seen someone fortunate enough to get such opportunities that wasn’t from affluence. Even the high school he attends (Basis International Park Lane Harbour^) is in an affluent resort community in southern China.
I don’t say this out of spite, it’s awesome he’s been able to get such opportunities and has been able to utilize them, but I think it’s important to be realistic about what these narratives imply without additional context and the answer is, sure, you can be successful with grit, brains and determination, but…the odds are very much not in your favor if you are in the wrong income bracket.
^ This is not doxxing. His LinkedIn profile states it plainly.
I don't know why all the posts fail to summarize the results properly.
I had a similar idea at the back of my head but here is a layman explanation:
Standard attention threads the previous layers output to the next layers input. By adding residual connections to each layer, the layers learn an update rule.
There is an obvious limitation here. Only the first layer gets to see the original input and all subsequent layers only get to see the previous layer output.
With attention residuals, the idea is that you have a tiny attention operator that decides between using the original input and any of the previous layer outputs.
21 comments
[ 2.9 ms ] story [ 43.2 ms ] thread1. Drops compute required for training by ~20%. This approach wont just help the ever escalating model sizes larger companies are pushing for, it means things like autoresearch can iterate on new model architectures faster.
2. WAY lower bandwidth requirements for inference. Means with approaches like this it should run on consumer hardware far better. It apparently requires 1/6th the memory bandwidth of a traditional approach for better results.
This is a big improvement if it can be generalized. They're claiming it's a drop in replacement, so it seems like it can as well.
> Abstract: Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. [...]
> Full AttnRes is straightforward but requires O(Ld) memory at scale. Block AttnRes partitions layers into N blocks, accumulates within each block via standard residuals, and applies attention only over block-level representations. With ~8 blocks, it recovers most of Full AttnRes's gains while serving as a practical drop-in replacement with marginal overhead.
Sadly, same can't be said about India (infrastructure/food security lags China).
In China since this class doesn't exist and forced busywork for class signaling is largely driven underground due to government policies driven young people just work on whatever the government policy says, which right now is tech. Hence you see smart young people getting into tech. While in the US they do Finance (if they built up the right portfolio to be allowed in the elite jobs club), speedrunning (if they didn't). Ironically the current US look a lot like imperial China, while China looks like the US of the late 1800s, but fully vertically integrated
So this is what really unsettles me. Not that China graduates more engineers every year than we have entirely employed in the US, but rather, that these individuals are not about delegating work, but actually doing it. Whereas the western credo is to get someone else to do the work (or in the words of PAtton, to get some one else to die for his country), I get the feeling that China will get robots and AI to do the work. I am reminded of the joke about Chinese factories having only 1 security guard and 1 dog. The guard is there to feed the dog.
I don’t say this out of spite, it’s awesome he’s been able to get such opportunities and has been able to utilize them, but I think it’s important to be realistic about what these narratives imply without additional context and the answer is, sure, you can be successful with grit, brains and determination, but…the odds are very much not in your favor if you are in the wrong income bracket.
^ This is not doxxing. His LinkedIn profile states it plainly.
I had a similar idea at the back of my head but here is a layman explanation:
Standard attention threads the previous layers output to the next layers input. By adding residual connections to each layer, the layers learn an update rule.
There is an obvious limitation here. Only the first layer gets to see the original input and all subsequent layers only get to see the previous layer output.
With attention residuals, the idea is that you have a tiny attention operator that decides between using the original input and any of the previous layer outputs.
Is it guaranteed to have the same effect on vanishing gradients though? What if it put weight 1 on a layer that had a tiny gradient?