And in 2029 people would call 10 trillion params models "medium size"...
I mean, with about $120K, we can technically run a 137B model locally. Any small company or rich individuals can afford this easily. So the "outrageous" back then is pretty "middle" now.
I can run a 30B parameter model on my M1 laptop at a pretty decent token rate. I can run two instances of the same model in parallel without affecting the speed.
A maxed out M2 Mac Studio can easily run 70B models and very likely 137B models too. The key is quantization.
So it’s all within reach for under $7k if you’re willing to accept a slightly lower quality (~5 bit quantization).
> Previous State-of-the-Art: [...] The number of parameters in the LSTM layers of these models vary from 2 million to 151 million.
> We present model architectures in which a MoE with up to 137 billion parameters
Back in 2017 most models were well under 1B, GPT2 (2019) was one of the first "big" non-MOE models at 1.5B in size. People weren't sure how well/much they would scale.
The CoralAI TPU has a mere 8 MB of SRAM in 2019!
GPT3 was 175B in 2020.
Now nearly all LLM's are at minimum 1B, but dense 70B is now common.
It also is a good reminder to revisit a lot of ideas and to contextualize many works appropriately. We've seen that __in general__, independent of architecture, model output quality increases as model parameters scale and data scales; under the assumption that data quality is sufficiently good and does not degrade with scale (quality of data is exceptionally important).
I find that this a common misinterpretation of a lot of papers and works done in the research community, especially by but far from exclusive to practitioners. There's a trend that big companies will simply out compute other models/methods and those results will be taken with the premise that the architecture is better. But if your model is only better because you out hyper-parameter tuned your model compared to another work, is your model actually better? We've seen extremely strong evidence that even the research community buys into hype as we've still seen that CNNs, when using training techniques similar to ViTs and similar parameter counts, perform just as well as transformer based models.
We likely leave a lot of potentially valuable models and architectures to rot because we don't properly contextualize our reading of works. I'd love to see universities without big tech partners explore new generative models but it's hard for them to pass review when rejection is as simple as "performs worse than model 1000x its size that uses massive pretraining and cost $2m to train," "not enough datasets to be convincing," (different from "needs x,y,z datasets to properly explore x',y',z' domains") or "but does it scale?" These all are "pay to play" responses and I think if anything, we've seen that the big boom in ML has actually been from letting people "fuck around and find out." But my main concern is we're pushing harder towards playing around with pretrained models (which are often proprietary) rather than doing this but also exploring new techniques. Context is everything and there are big differences between a paper attempting to be SOTA and a work trying to explore different ideas (see NASA TRL). It's very easy to get caught up in the hype and lose sight, because evaluation is an exceptionally difficult task (if it were easy, we'd let LLMs review, but please for the love of god, no).
In response to a very impressive set of LLM open weights released today from Mistral AI called "Mixtral 8x7B", I was reminded of this amazing publication on the origin of "sparse mixture of experts" from none other than Geoffrey Hinton and Jeff Dean.
The Sparse Mixture of Experts neural network architecture is actually an absolutely brilliant move here.
It scales fantastically, when you consider that (1) GPU RAM is way too expensive, in financial dollars, (2) SSD / CPU RAM are relatively cheap, and (3) you can have "experts" running on their own computers, i.e. it's a natural distributed computing partitioning strategy for neural networks.
I did my M.S. thesis on large-scale distributed deep neural networks in 2013 and can say that I'm delighted to point out where this came from.
In 2017, it emerged from a Geoffrey Hinton / Jeff Dean / Quoc Le publication called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer".
Here is the abstract: "The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost."
So, here's a big A.I. idea for you: what if we all get one of these sparse Mixture of Experts (MoEs) that's a 100 GB on our SSDs, contains all of the "outrageously large" neural network insights that would otherwise take specialized computers, and is designed to run effectively on a normal GPU or even smaller (e.g. smartphone)?
11 comments
[ 3.0 ms ] story [ 20.6 ms ] threadI mean, with about $120K, we can technically run a 137B model locally. Any small company or rich individuals can afford this easily. So the "outrageous" back then is pretty "middle" now.
A maxed out M2 Mac Studio can easily run 70B models and very likely 137B models too. The key is quantization.
So it’s all within reach for under $7k if you’re willing to accept a slightly lower quality (~5 bit quantization).
> We present model architectures in which a MoE with up to 137 billion parameters
Back in 2017 most models were well under 1B, GPT2 (2019) was one of the first "big" non-MOE models at 1.5B in size. People weren't sure how well/much they would scale.
The CoralAI TPU has a mere 8 MB of SRAM in 2019!
GPT3 was 175B in 2020.
Now nearly all LLM's are at minimum 1B, but dense 70B is now common.
I find that this a common misinterpretation of a lot of papers and works done in the research community, especially by but far from exclusive to practitioners. There's a trend that big companies will simply out compute other models/methods and those results will be taken with the premise that the architecture is better. But if your model is only better because you out hyper-parameter tuned your model compared to another work, is your model actually better? We've seen extremely strong evidence that even the research community buys into hype as we've still seen that CNNs, when using training techniques similar to ViTs and similar parameter counts, perform just as well as transformer based models.
We likely leave a lot of potentially valuable models and architectures to rot because we don't properly contextualize our reading of works. I'd love to see universities without big tech partners explore new generative models but it's hard for them to pass review when rejection is as simple as "performs worse than model 1000x its size that uses massive pretraining and cost $2m to train," "not enough datasets to be convincing," (different from "needs x,y,z datasets to properly explore x',y',z' domains") or "but does it scale?" These all are "pay to play" responses and I think if anything, we've seen that the big boom in ML has actually been from letting people "fuck around and find out." But my main concern is we're pushing harder towards playing around with pretrained models (which are often proprietary) rather than doing this but also exploring new techniques. Context is everything and there are big differences between a paper attempting to be SOTA and a work trying to explore different ideas (see NASA TRL). It's very easy to get caught up in the hype and lose sight, because evaluation is an exceptionally difficult task (if it were easy, we'd let LLMs review, but please for the love of god, no).
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer - https://news.ycombinator.com/item?id=13518039 - Jan 2017 (81 comments)
Outrageously large neural networks: the sparsely-gated mixture-of-experts layer - https://news.ycombinator.com/item?id=12963364 - Nov 2016 (2 comments)
The Sparse Mixture of Experts neural network architecture is actually an absolutely brilliant move here. It scales fantastically, when you consider that (1) GPU RAM is way too expensive, in financial dollars, (2) SSD / CPU RAM are relatively cheap, and (3) you can have "experts" running on their own computers, i.e. it's a natural distributed computing partitioning strategy for neural networks.
I did my M.S. thesis on large-scale distributed deep neural networks in 2013 and can say that I'm delighted to point out where this came from.
In 2017, it emerged from a Geoffrey Hinton / Jeff Dean / Quoc Le publication called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer".
Here is the abstract: "The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost."
So, here's a big A.I. idea for you: what if we all get one of these sparse Mixture of Experts (MoEs) that's a 100 GB on our SSDs, contains all of the "outrageously large" neural network insights that would otherwise take specialized computers, and is designed to run effectively on a normal GPU or even smaller (e.g. smartphone)?