Starcraft in the form of Alphastar worked in the sense that it could beat humans, at least in the short term. The problem with the whole technique is that they had to tether it to the human examples they had gathered in…
Interesting point. Nvidia have been improving the int performance for quantized inference on their GPUs a lot. It might be a lot of work but could it be possible to scale up this NNUE approach to the point where it…
The published hyperparameters are usually ridiculously conservative, for the simple games like breakout and pong you can usually converge in far fewer frames than in the papers.
> We're getting a bit off-topic here, but the #1 target by far in reducing greenhouse emissions is power generation. I readily admit I don't know any of the numbers associated with carbon production and my comment was…
This is a very valid argument but it's hard to know what scaling a transformer will really do without trying (looking at you GPT-3). This is probably an issue for ML in general at this point. I think a more nuanced…
Yes it's something I often see ignored as "common knowledge" dictates that in ML inference is way cheaper than training. But if you're running a model in production at google with loads of google searches hitting it…
Yann is a consistently sober voice in this world of AI hype. I find it quite refreshing. Personally I see little evidence that this "just scale a transformer until sentience" hype-train is going to take us anywhere…
> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the…
I would start with David Silvers (DeepMind) youtube series to get an idea of what's possible or not. Running an already trained reinforcement learning agent is relatively cheap (unless your model is massive). I suspect…
It depends on how intensive your pre-processing pipeline is. With a really fast accelerator you can quite easily start to be bottlenecked by your CPU.
Vectorization: I'm not an expert in this area so I can only tell you what I've personally found difficult in dealing with vectorization. Usually it all comes down to alignment and vector lanes. To utilize the vector…
Bias is just a scalar term that is added. You can learn it via backpropagation like all the other weights.
Since you are right that I have no idea what you are talking about, could you explain what you are talking about?
I find it very surprising that someone would rely on unvalidated mathematical models for this, that goes for the Imperial College people as well as Sweden. Are they even able to fit the parameters in retrospect? Anyone…
That article doesn't state that 14% of cases were asymptomatic.
I'm not a mathematician but coming from the software world if one guy wrote a massive program (I'm assuming 600 pages is massive) in "an impenetrable, idiosyncratic style" you could virtually guarantee it would not be…
Starcraft in the form of Alphastar worked in the sense that it could beat humans, at least in the short term. The problem with the whole technique is that they had to tether it to the human examples they had gathered in…
Interesting point. Nvidia have been improving the int performance for quantized inference on their GPUs a lot. It might be a lot of work but could it be possible to scale up this NNUE approach to the point where it…
The published hyperparameters are usually ridiculously conservative, for the simple games like breakout and pong you can usually converge in far fewer frames than in the papers.
> We're getting a bit off-topic here, but the #1 target by far in reducing greenhouse emissions is power generation. I readily admit I don't know any of the numbers associated with carbon production and my comment was…
This is a very valid argument but it's hard to know what scaling a transformer will really do without trying (looking at you GPT-3). This is probably an issue for ML in general at this point. I think a more nuanced…
Yes it's something I often see ignored as "common knowledge" dictates that in ML inference is way cheaper than training. But if you're running a model in production at google with loads of google searches hitting it…
Yann is a consistently sober voice in this world of AI hype. I find it quite refreshing. Personally I see little evidence that this "just scale a transformer until sentience" hype-train is going to take us anywhere…
> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the…
I would start with David Silvers (DeepMind) youtube series to get an idea of what's possible or not. Running an already trained reinforcement learning agent is relatively cheap (unless your model is massive). I suspect…
It depends on how intensive your pre-processing pipeline is. With a really fast accelerator you can quite easily start to be bottlenecked by your CPU.
Vectorization: I'm not an expert in this area so I can only tell you what I've personally found difficult in dealing with vectorization. Usually it all comes down to alignment and vector lanes. To utilize the vector…
Bias is just a scalar term that is added. You can learn it via backpropagation like all the other weights.
Since you are right that I have no idea what you are talking about, could you explain what you are talking about?
I find it very surprising that someone would rely on unvalidated mathematical models for this, that goes for the Imperial College people as well as Sweden. Are they even able to fit the parameters in retrospect? Anyone…
That article doesn't state that 14% of cases were asymptomatic.
I'm not a mathematician but coming from the software world if one guy wrote a massive program (I'm assuming 600 pages is massive) in "an impenetrable, idiosyncratic style" you could virtually guarantee it would not be…