I got a degree in applied mathematics. Some of my favorite courses were in pure math. I think I came away with my degree better for having taken the detour. My only regret at graduation was that I hadn't taken more.
> “Modern machine learning problems often involve fitting a huge number of parameters with a massive amount of data. GPT-4, the next iteration of the engine underlying ChatGPT, is rumored to have 1 trillion to 100 trillion parameters. No computer in existence could handle these parameters all at once. So in each step, algorithms pick a small random subset of parameters (whatever the computer can handle) and just work with those instead.”
I can’t wrap my head around this - is this an oversimplification? Does an input take some n runs and then the outputs are summed? Since each hidden layer has a connection to every neuron in the previous hidden layer, how are the results not severely warped?
(I’m having trouble articulating my non-understanding, sorry for the low quality question. Perhaps if someone has a link to a paper on this technique that could be helpful)
- Details about GPT4 aren't very public. That's likely just a guess as to what's happening.
- 100T parameters isn't actually that hard to use on modern hardware. Getting it right with accelerators in the mix is a bit more challenging, but the assumption is exaggerated.
- An analogous idea in the literature is "dropout". So long as the network is trained with that same behavior it's usually beneficial. Dropping weights like the article describes is akin to severing connections in the graph of neuron connections. Traditional dropout ignores outputs at a given layer, which is akin to severing a whole neuron's worth of connections.
- Some intuition about the lack of warping is that dropout is equivalent to applying the dense transformation, dividing by that probability (to undo the skewed mean), and adding some noise. Training with dropout produces a network robust to noise (sort of like how tanh(100x) strongly reduces noise to push outputs toward a particular behavior -- compressing large domains into small ranges). Applying noise at inference time is atypical (usually the network is fully applied and then rescaled by that probability to have deterministic outputs fully using all weights), but it shouldn't yield drastically worse results.
- They'll likely get better results at inference time if, if they're averaging (might just do one sparse pass and call it a day) they do so at each layer. It helps reduce the chance of noise amplifying through the network and isn't any more expensive to compute (in any reasonable implementation it'd have lower communication costs, so should be a bit faster).
Thanks for this comment. I’m going to spend some time looking into whitepapers on dropout - this is a new technique to me. Thanks for the in depth comment, I appreciate it a bunch.
6 comments
[ 3.2 ms ] story [ 20.8 ms ] thread"In subsequent steps of the algorithm, should it pick parameters that we have already picked before in earlier steps, or should we exclude those?"
Isn't it the training examples the two different types of sampling are considered for, instead of the parameters of the model?
I can’t wrap my head around this - is this an oversimplification? Does an input take some n runs and then the outputs are summed? Since each hidden layer has a connection to every neuron in the previous hidden layer, how are the results not severely warped?
(I’m having trouble articulating my non-understanding, sorry for the low quality question. Perhaps if someone has a link to a paper on this technique that could be helpful)
- Details about GPT4 aren't very public. That's likely just a guess as to what's happening.
- 100T parameters isn't actually that hard to use on modern hardware. Getting it right with accelerators in the mix is a bit more challenging, but the assumption is exaggerated.
- An analogous idea in the literature is "dropout". So long as the network is trained with that same behavior it's usually beneficial. Dropping weights like the article describes is akin to severing connections in the graph of neuron connections. Traditional dropout ignores outputs at a given layer, which is akin to severing a whole neuron's worth of connections.
- Some intuition about the lack of warping is that dropout is equivalent to applying the dense transformation, dividing by that probability (to undo the skewed mean), and adding some noise. Training with dropout produces a network robust to noise (sort of like how tanh(100x) strongly reduces noise to push outputs toward a particular behavior -- compressing large domains into small ranges). Applying noise at inference time is atypical (usually the network is fully applied and then rescaled by that probability to have deterministic outputs fully using all weights), but it shouldn't yield drastically worse results.
- They'll likely get better results at inference time if, if they're averaging (might just do one sparse pass and call it a day) they do so at each layer. It helps reduce the chance of noise amplifying through the network and isn't any more expensive to compute (in any reasonable implementation it'd have lower communication costs, so should be a bit faster).