I'm really excited to see folks starting to talk about parallelizing machine learning. The conversation has been dominated by GPU-friendly techniques - a classic example of "everything looks like a nail when you have a hammer".
I hope we start seeing more massively parallel training strategies (most likely with GPUs under the hood still)
This is a strange comment to see. The techniques in TFA have been at the heart of a the last few years of progress in large-scale language models, image generation, etc?
Dude, parallelization is exactly how non-trivial neural networks are trained since the beginning.
Parallel processing as in SIMD architecture is what makes Deep Learning possible now.
Parallel computing as in stacking graphics card is how literally everything is done everywhere- from big tech companies to not-so-well-funded data science departments in unknown unis.
Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!
2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).
> In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated).
In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck?
What does it mean then for to have a high "computation per weight".
>• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations.
• Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations.
Wouldn't fc's require more computation since each parameter has more incoming and outgoing values?
I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!
Convolutional kernels are matrix-multiplied over and over again in a sliding window across the feature map. The fully connected layer is matrix-multiplied exactly once. So the number of times each element of the weight matrix is used depends on the size of the layer for FC, but depends on the size of the feature map for Conv layers. Usually the feature map is much larger than the kernel, for the majority of layers, so the kernel weights are re-used a lot.
The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.
The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)).
Otherwise the peer comment answered your question really well!
"One weird trick" is still pretty much the first way to go for most recommendation systems that are large embeddings focused. see torchrec, nvidia's hugectr
I was curious as to what the 'community AI' research org's stances on distributed training of deep neural nets were so some weeks ago I stumbled upon Eleuther AI's FAQ page which was talking about how it was not a task that they were looking at due to various technological challenges:
What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind?
-Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient.
-Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters.
-Having enough contributors to outweigh the high overhead is infeasible.
-Verifiability and resistance to outside attack are not currently possible without significant additional overhead.
In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.
---
Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.
23 comments
[ 3.0 ms ] story [ 62.7 ms ] threadI hope we start seeing more massively parallel training strategies (most likely with GPUs under the hood still)
This is a strange comment to see. The techniques in TFA have been at the heart of a the last few years of progress in large-scale language models, image generation, etc?
Parallel processing as in SIMD architecture is what makes Deep Learning possible now.
Parallel computing as in stacking graphics card is how literally everything is done everywhere- from big tech companies to not-so-well-funded data science departments in unknown unis.
You are so so off.
https://openai.com/blog/evolution-strategies/
Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!
http://pchiusano.github.io/2019-07-06/learning-without-a-gra...
http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
I think you’re right that the area is underexplored.
CMA-ES: https://blog.otoro.net/2017/10/29/visual-evolution-strategie...
ARS: https://arxiv.org/abs/1803.07055
Those optimizers are efficient up to certain neural network sizes.
2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).
> In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated).
In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck?
What does it mean then for to have a high "computation per weight".
>• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations. • Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations.
Wouldn't fc's require more computation since each parameter has more incoming and outgoing values?
I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!
The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.
Otherwise the peer comment answered your question really well!
What’s different?
Source: https://www.eleuther.ai/faq/
What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind? -Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient. -Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters. -Having enough contributors to outweigh the high overhead is infeasible. -Verifiability and resistance to outside attack are not currently possible without significant additional overhead. In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.
---
Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.