HN Explorer

6 comments

[ 3.6 ms ] story [ 24.8 ms ] thread
sabalaba
Stephen from Lambda here. These are, to our knowledge, the first public Deep Learning benchmarks for the 2080 Ti on actual hardware. We've provided all of the code and tooling to reproduce these results yourself so, if you have a 2080 Ti or another GPU you want to benchmark, please feel free to reproduce these results yourself:

https://github.com/lambdal/lambda-tensorflow-benchmark

wmf
Were tensor cores used?
sabalaba
yes
wmf
Interesting. I would have expected FP16 with tensor cores to give at least a 100% speedup, not just 60%.
nabla9
Thank you.

You compared each model fixed number of times (images/sec). What was the training accuracy difference between FP32 and FP16? Do they match?

I would like find out the efficiency gain when training he same model to same accuracy with different floating points types and the same model size.

sabalaba
The training accuracy won't match. Here's an excerpt from a presentation on FP16 training to give you an idea of the loss in accuracy:

    Mode                     | Top1 accuracy, % | Top5 accuracy, %
    -------------------------+------------------+---------------------
    Fp32                     | 58.62            | 81.25
    Mixed precision training | 58.12            | 80.71
    FP16 training            | 54.89            | 78.12
[1] http://on-demand.gputechconf.com/gtc/2017/presentation/s7218...