I think gp's referring to this meme [1] embedded in the article. I believe a charitable interpretation would be that it's satirising the various entities who have voiced opinions against large scale deep learning (from skeptics and theorists, to social activist types that want to slow or stop DL's march). No need to take it too seriously.
H.He has some overlap with the EAI community which has more crab-bucket tendencies when picking topics and/or mocking other groups. Even the more positive sentiments seem to be somehow crab-bucketed a bit (and, bizarrely at least to me, a large amount of politicking).
I'll visit there occasionally, but it's just too toxic a sphere for me to want to contribute research to. I get an itchy feeling whenever I try to seriously investigate an opportunity for contributing open source research in that group.
Great point about Sutton's law.
I always use Hinton's Capsule Networks as an example of an algorithm that may work in theory, but doesn't scale on existing hardware, due it's voting stage, which is not parallelizable on existing hardware accelerators.
That's a great point, I hadn't really thought about that a lot, and definitely had not made the hard connection.
I do love how he continues to spin out new potential modalities for Deep Learning. While underrecognized due to the hype today, I think we darn well need far more of it these days! :D :))))
> Experiment noise has taught me so much over several thousands of manually-run (yes, I'm a masochist, it's a personal preference) experiments converging in a few seconds.
Is there any resource that makes legible how to go about this? Or shares insights into the process of iterative learning through experiments, more broadly?
I'd like to create it in the future, but I can share a rough version of what I have. Basically, as long as you gain more information than noise by scaling your experiments down (i.e., the smaller models create similar directions in the needle), then it's easier to run lots of experiments, and much more cheaply oftentimes too. Remember that performance oftentimes runs on a log(1+p) curve in terms of cost/time/complexity vs reward.
If you write your codebases to minimize the expected time to make a particular change (fewer files, simpler proxy problem [as long as it transfers to your larger-scale problem!], more 'flat' structure, simple dataloaders, etc), then that is extremely valuable as well. What you are optimizing is the average time from idea-to-answer.
What is nice is that experiments go from strongly-planned, biased (from human beliefs, etc) things to a more high-variance process where there isn't as much requirement to double-and-triple check everything before running it. There is often an _extremely_ poor mismatch between the very first impression of what many of us think will work and will not and what actually works when the dust settles. Sorta like how SGD bounces around the loss landscape before settling in.
One thing that also is easier with fast experiments is to make the 'opposite' change if a planned change doesn't work out well, just to see how it goes. A tiny thing, but if you're going sequentially instead of batched (in terms of batched experiments) it can be useful.
I use as few libraries as possible. A lot of the bulky monitoring solutions can be good for some enterprise things, I don't find they're good for innovating, but instead, integrating. You innovate on small test problems that are appropriately representative. When done, you integrate and see what difference it makes.
Doing dumb things is one of the best ways to learn. Having access to everything to be able to print it out/log it/look at it in a chart is great too.
But everything is bottlenecked by speed. Increase your experiment speed, where (critically!) your speed includes the time from idea to first answer coming back (and not just plain ol' runtime), and you'll find oftentimes it's easy to get 5-10x research speed improvements. We just don't do research all that efficiently these days, I don't think. But we can. And I'm sure we will, eventually. :3
There are a few things like scaling that can be difficult, but math is still math, and if you pick your proxies right, it should be an explored-translation-step after the proxy problem behaves as well as it is reasonably able. And if you have a secondary, slightly larger proxy problem, that's an even better thing in my experience, since it prevents the sticker shock of broken graphs and things failing for an unknown reason. All about how well you can pass the peach from one person to another. :')))) <3 <3 :')))) :'D :'D
Tysm; this is good stuff for me to ruminate on. One of the challenges for me has been switching to this very empirical orientation (I’m far more used to a conceptual understanding driven mode). If & when you do get around to fleshing out your perspective, I think it would be quite invaluable :-)
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people.
It's just a meme. No need to read too much into it.
In this case, the meme came from https://gwern.net/scaling-hypothesis. I doubt that Gwern was making fun of any particular group, except perhaps those that disagree with the bitter lesson.
The fluidity of language is one of the most fascinating aspects of it! English isn't English. It's much more narrowly defined with a geographic and temporal context. Meanings of words and phrases shift over time. Idioms come and origins are forgotten, but it's all folded into cultural dialects. Each generation is able to have their own mark on language, and there's probably a story told in the evolution of the English language across time that maps out the cultural and geographic changes as well. Now I'm curious what sort of research has been done into this. Regardless, the point is language changes. No one can stop it. Raging against it just reveals that you're old and out of touch. After all, I doubt those people raged against the slang of their youth. It's always the Next Generation that is The Problem.
> As an aside and a comment on the article, I feel dismayed by the author's decision to include the 'go brrr' meme. It's connected with toxic sentiment and communities from where it's generally used, and is oftentimes used to needlessly mock a group of people. There was good content in reading the article, but it turned me off at the beginning as it was rather childish -- in not-a-pleasant way -- and unfunny, and it made the rest of the article hard to read with that taste in my mouth.
I have questions...
Is the author aware of the toxic communities and their use of the meme or is the author a member of these toxic communities?
Is it the authors intent to exclude you in anyway when using the meme?
And if there a bunch of "no" answers above: Why is it the authors responsibility to keep track with your associations with this meme?
I did address those earlier in terms of association, I'm not sure if I understand the aggressive tone of this response however, or of a few of the assumptions in it. A few other people have made similar points about this culture as well.
I think your comment comes off as "concern trolling" whether you intended it to be or not. And indeed we are now off the topic and discussing "going brr" instead of the subject.
That's a fair response. I'm generally okay (some modern connotations aside) with people using the going brr bit. The servers where H.He spends a lot of time in actively use the meme when mocking outside groups (w/ few neutral uses), and it's a bit frustrating as it actively causes rifts that make it harder for me to do professional work.
I bring it up because I'd like to bring light on it if it's sorta seeping into the mainstream, a lot of people adjacent to it sort of just ignore it and leave it be. Like, I've actively seen this meme used in the main AI servers where it's used to push down people who do question blind scaling, for example (which is not what Sutton was talking about). It silences conversation and halts diversity, and it's an actual problem unfortunately.
Thankfully there are other spaces where it's not as much a problem, but since it's pretty well-tied and trickling into HN, felt appropriate to share my 2c. And people may well say what they want (including brr) in response. :)
I'm trying to get the point across that not all people come from the same place and don't keep the same connotations and associations with the memes they use. Making them responsible for your associations is in my view fairly aggressive.
If you feel validated by people that think the same thing go ahead, but I think is that the 'meme policing' was unnecessary on a technical response of the content.
This feels pretty aggressive, I think we're on a different wavelength here. That's not quite what I'm saying, but it seems best for us to leave it at that.
Sorry but I'm actively trying to help you here. From the other reply
> it's a bit frustrating as it actively causes rifts that make it harder for me to do professional work.
My comments that you are taking that I'm being aggressive is that you have made your professional frustration about your feelings. These issues isn't who else uses the meme, but that the meme spreads a bad idea that makes your professional life harder.
26 comments
[ 2.7 ms ] story [ 70.5 ms ] thread[1] https://horace.io/img/perf_intro/gpus_go_brrr.webp
I'll visit there occasionally, but it's just too toxic a sphere for me to want to contribute research to. I get an itchy feeling whenever I try to seriously investigate an opportunity for contributing open source research in that group.
I do love how he continues to spin out new potential modalities for Deep Learning. While underrecognized due to the hype today, I think we darn well need far more of it these days! :D :))))
Is there any resource that makes legible how to go about this? Or shares insights into the process of iterative learning through experiments, more broadly?
If you write your codebases to minimize the expected time to make a particular change (fewer files, simpler proxy problem [as long as it transfers to your larger-scale problem!], more 'flat' structure, simple dataloaders, etc), then that is extremely valuable as well. What you are optimizing is the average time from idea-to-answer.
What is nice is that experiments go from strongly-planned, biased (from human beliefs, etc) things to a more high-variance process where there isn't as much requirement to double-and-triple check everything before running it. There is often an _extremely_ poor mismatch between the very first impression of what many of us think will work and will not and what actually works when the dust settles. Sorta like how SGD bounces around the loss landscape before settling in.
One thing that also is easier with fast experiments is to make the 'opposite' change if a planned change doesn't work out well, just to see how it goes. A tiny thing, but if you're going sequentially instead of batched (in terms of batched experiments) it can be useful.
I use as few libraries as possible. A lot of the bulky monitoring solutions can be good for some enterprise things, I don't find they're good for innovating, but instead, integrating. You innovate on small test problems that are appropriately representative. When done, you integrate and see what difference it makes.
Doing dumb things is one of the best ways to learn. Having access to everything to be able to print it out/log it/look at it in a chart is great too.
But everything is bottlenecked by speed. Increase your experiment speed, where (critically!) your speed includes the time from idea to first answer coming back (and not just plain ol' runtime), and you'll find oftentimes it's easy to get 5-10x research speed improvements. We just don't do research all that efficiently these days, I don't think. But we can. And I'm sure we will, eventually. :3
There are a few things like scaling that can be difficult, but math is still math, and if you pick your proxies right, it should be an explored-translation-step after the proxy problem behaves as well as it is reasonably able. And if you have a secondary, slightly larger proxy problem, that's an even better thing in my experience, since it prevents the sticker shock of broken graphs and things failing for an unknown reason. All about how well you can pass the peach from one person to another. :')))) <3 <3 :')))) :'D :'D
It's just a meme. No need to read too much into it.
In this case, the meme came from https://gwern.net/scaling-hypothesis. I doubt that Gwern was making fun of any particular group, except perhaps those that disagree with the bitter lesson.
I thought it meant to involuntary shiver...
I have questions...
Is the author aware of the toxic communities and their use of the meme or is the author a member of these toxic communities? Is it the authors intent to exclude you in anyway when using the meme?
And if there a bunch of "no" answers above: Why is it the authors responsibility to keep track with your associations with this meme?
I bring it up because I'd like to bring light on it if it's sorta seeping into the mainstream, a lot of people adjacent to it sort of just ignore it and leave it be. Like, I've actively seen this meme used in the main AI servers where it's used to push down people who do question blind scaling, for example (which is not what Sutton was talking about). It silences conversation and halts diversity, and it's an actual problem unfortunately.
Thankfully there are other spaces where it's not as much a problem, but since it's pretty well-tied and trickling into HN, felt appropriate to share my 2c. And people may well say what they want (including brr) in response. :)
If you feel validated by people that think the same thing go ahead, but I think is that the 'meme policing' was unnecessary on a technical response of the content.
> it's a bit frustrating as it actively causes rifts that make it harder for me to do professional work.
My comments that you are taking that I'm being aggressive is that you have made your professional frustration about your feelings. These issues isn't who else uses the meme, but that the meme spreads a bad idea that makes your professional life harder.