From first looks, there is little doubt that that NVIDIA's Volta architecture is a monster and will revolutionize the AI and HPC market. But the article seems to avoid quantifying how 16-point FP operations are beneficial against 32- or 64-bit FP operations in real-world usecases, or how the Caffe2 / NVIDIA architecture provides any significant boost to FP16 in particular, especially apropos to images (or why FP16 is better for image data in general).
I'm interested more in understanding why Caffe2 would outperform Theano, Tensorflow, MXNet, etc. once Volta chipsets are generally available, beyond early pre-release optimization, particularly when most of the front-runners are already leveraging / taking into account NCCL, CuDNN, NVLink, etc. When the burden of adding support for new NVIDIA primitives is so low, what gives Caffe2 an advantage beyond an ephemeral "we were partners with NVIDIA first" one-up that would last for a couple of months at most?
(Apologies in advance if this post sounds overly negative, but I am constantly evaluating the current crop of frameworks for the trade-offs they enforce on the problem space, and a definitive answer would be very helpful.)
Caffe2 seems to be engineered for efficiency as opposed to flexibility. For example, Caffe2 is apparently Facebook's go-to choice for mobile edge device deployment for applications such as style transfer.
I mean, it uses a static graph definition instead of dynamic graphs (allows for deployment optimizations), but is there anything Caffe2-specific that would give it a leg up over Theano/TensorFlow/whatever other static graph library?
16 bit FP operations seem to be more than enough for most networks - in fact for inference, google claims in their TPU paper that most network parameters can be quantized down to 8 integer bits!
There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.
Most desktop chips offer little advantage for FP16 (at best that the source memory footprint is smaller, maybe better cache hit rates), but newer GPUs can actually blaze. If it has the magnitude/precision needs for a task, it's a big win.
FP16 multipliers are smaller (in terms of silicon footprint), which means that you can fit more of them on a given silicon die, and they use less power than FP32. The other advantage is that FP16 uses half as much memory, meaning that FP16 operations require half as much memory traffic, which can lead to faster computations. It can also allow you to fit models on your GPU which have twice as many parameters.
The reason you don't typically have FP16 on CPUs is that in the CPU world, cache dominates die area and power usage. The floating point ALU component of a CPU is relatively tiny. Still, even in the CPU world, Intel recently added instructions to convert to/from FP16 before operating on it, because they know there is a memory bandwidth advantage in using a smaller floating-point format.
The GPU looks like a monster, and I am sure it'll deliver more power to AI applications, but what I really want to know is when I can put one in my gaming PC.
I think it's great that what started as a specialist gaming device is now being used in industry for Big Things. The development cost that Nvidia et al. have invested in new designs has undoubtedly been financed in (large) part by the gaming community. Now income and advancements for both sectors feed into the other and gamers like me are reaping the benefits with reduced price:performance across the range.
Meanwhile I still remember being seated at the Games Development Conference talk in 2009 where Intel tried to convince us how Larrabee would change the world of computing.
Still waiting for them to produce anything worthwhile buying instead of AMD and NVidia GPUs.
Their main sales pitch is "given that it's a bunch of x86 put together, you don't have to port your code to get Massive Paralellization by Intel (TM)". Some supercomputers use them, but it's true: I've never seen a worthwhile comparisson between that and a strong (or equivalent) GPGPU.
There are some very interesting talks by Intels experts on how to use these guys with AVX and AVX512 (or however its called now) and the multiplicative improvement both multicore and AVX give when used together (the sales line goes along like this "10x for multicore, and on top of that 4x for AVX, voilá 40x speedup!").
I don't work in an HPC environment, so I don't really know if they stand up in reality to Intels claims.
These efforts are mostly going to aid further market segmentation and boosting profits; e.g. even for development you'll now need a Tesla/Quadro that costs a ton of money, the days of GTX cards that have the same GK2xx or GM2xx chip as Teslas is over.
While the "tensor" units may land in consumer stuff (though I don't think it'll happen this year) I expect there to be little "trickle-down" planned or wanted at this stage.
The GTX 1070 for laptops is destined to be the most interesting price/performance opportunity for a while imho, you can even undervolt the CPU -0.100 to -0.150 to reduce overheating.
I was told Q4 2017 for non-graphics consumer (what kind of consumer are you referring to), but that is earlier than most others are saying. https://news.ycombinator.com/item?id=14310633
In consumer terms AMD's got a compelling product for mid-range GPUs where most of the money is but NVidia's got the top of the range locked up. Hopefully we'll see some good chips out of AMD soon which combine a genuinely good CPU core with a good GPU in one package which'll be really nice in the low end of the market.
As for scientific computing I suppose there are workloads where having your GPU and CPU on one die might give you latency savings that are worth it but mostly NVidia is going to keep owning that market for the next few years.
The mid-range has a lot of potential as it's based on Fiji which NVIDIA can't match in terms of memory BW with any of their existing ML products (nor near future ones). This is important IMO because ML inference is memory-bound and GDDR5 on NVIDIA's cards is no match for even 1st gen HBM.
The big cards is announced to be 12.5/25 Tflops SP/HP, so as long as it's not "tensoring" it should be in between P100 and V100. If they get the price right (and given that even P100 is currently still $5.5-6 w/o tax), it will hopefully gain traction.
A lot hinges on their fully OSS (!) software stack, but they are getting things done, just had a new release: https://rocm.github.io
I understand the pedigree of these cards, but at what point do we stop calling them GPUs and start calling them something else? MPU? TPU? I don't know, but isn't it a little bit weird to keep using the word "graphics" for something that is being made more and more specifically for other things?
Understood. I’m just commenting on the name in a vacuum. If it’s supposed to mean use for non-graphics calculations, it’s confusing to have “graphics” in the name.
No, it's several specific purposes that heavily use the provided general primitives. There's nothing stopping more things from using it, which makes it general.
You don't call your PC a specific purpose computer just because all you do is use google chrome.
You can still call it a GPU, but that now stands for General Processing Unit. It pairs well with CPU, being the Central Processing Unit in the center of the entire machine and GPUs being added for additional processing capability.
Surely getting people to refer to their vector lanes as "cores" was their biggest piece of marketing magic. Yes, they're more flexible than SIMD cores but not more so than an OoO core's execution ports.
It's interesting the speed up isn't more pronounced between Volta and Pascal considering the Tensor cores on paper give you about 6x the MFlops. The price differential looks large.
From AnandTech: "By the numbers, Tesla V100 is slated to provide 15 TFLOPS of FP32 performance, 30 TFLOPS FP16, 7.5 TFLOPS FP64, and a whopping 120 TFLOPS of dedicated Tensor operations. With a peak clockspeed of 1455MHz, this marks a 42% increase in theoretical FLOPS for the CUDA cores at all size. Whereas coming from Pascal, for Tensor operations the gains will be closer to 6-12x, depending on the operation precision."
52 comments
[ 2.9 ms ] story [ 30.6 ms ] threadEDIT: Nvidia's advertised TFLOPS are:
That's a core that does 4x4 FP16 matrix multiplication + 4x4 FP32 accumulation in one go.
That's where V100 gets its boost, up to 120 TFLOPS.
I'm interested more in understanding why Caffe2 would outperform Theano, Tensorflow, MXNet, etc. once Volta chipsets are generally available, beyond early pre-release optimization, particularly when most of the front-runners are already leveraging / taking into account NCCL, CuDNN, NVLink, etc. When the burden of adding support for new NVIDIA primitives is so low, what gives Caffe2 an advantage beyond an ephemeral "we were partners with NVIDIA first" one-up that would last for a couple of months at most?
(Apologies in advance if this post sounds overly negative, but I am constantly evaluating the current crop of frameworks for the trade-offs they enforce on the problem space, and a definitive answer would be very helpful.)
(In fact there are even some experiments with a very low number of bits - can't find the link now though)
There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.
A vendor-made benchmark where the vendor is outperforming other vendors.
Let's not jump to conclusion on what's really fastest.
https://dennisforbes.ca/index.php/2017/04/11/floating-point-...
Most desktop chips offer little advantage for FP16 (at best that the source memory footprint is smaller, maybe better cache hit rates), but newer GPUs can actually blaze. If it has the magnitude/precision needs for a task, it's a big win.
The reason you don't typically have FP16 on CPUs is that in the CPU world, cache dominates die area and power usage. The floating point ALU component of a CPU is relatively tiny. Still, even in the CPU world, Intel recently added instructions to convert to/from FP16 before operating on it, because they know there is a memory bandwidth advantage in using a smaller floating-point format.
I think it's great that what started as a specialist gaming device is now being used in industry for Big Things. The development cost that Nvidia et al. have invested in new designs has undoubtedly been financed in (large) part by the gaming community. Now income and advancements for both sectors feed into the other and gamers like me are reaping the benefits with reduced price:performance across the range.
Still waiting for them to produce anything worthwhile buying instead of AMD and NVidia GPUs.
There are some very interesting talks by Intels experts on how to use these guys with AVX and AVX512 (or however its called now) and the multiplicative improvement both multicore and AVX give when used together (the sales line goes along like this "10x for multicore, and on top of that 4x for AVX, voilá 40x speedup!").
I don't work in an HPC environment, so I don't really know if they stand up in reality to Intels claims.
While the "tensor" units may land in consumer stuff (though I don't think it'll happen this year) I expect there to be little "trickle-down" planned or wanted at this stage.
How are things in the red camp ? There was some HIP thing where Fiji was as good as Pascal.
Wait for ryzen.
Wait for vega.
Wait for ryzen again.
Vega is coming out this year. Volta isn't consumer based. Go check the enterprise prices.
Consumer Volta is mid 2018, so this, from a consumer perspective, is a paper launch.
Also, recent Linux patches hint that there will be dual-chip water-cooled Vega boards too which is likely not even 1080 Ti regime: https://www.techpowerup.com/233208/linux-drivers-point-to-up...
As for scientific computing I suppose there are workloads where having your GPU and CPU on one die might give you latency savings that are worth it but mostly NVidia is going to keep owning that market for the next few years.
The mid-range has a lot of potential as it's based on Fiji which NVIDIA can't match in terms of memory BW with any of their existing ML products (nor near future ones). This is important IMO because ML inference is memory-bound and GDDR5 on NVIDIA's cards is no match for even 1st gen HBM.
The big cards is announced to be 12.5/25 Tflops SP/HP, so as long as it's not "tensoring" it should be in between P100 and V100. If they get the price right (and given that even P100 is currently still $5.5-6 w/o tax), it will hopefully gain traction.
A lot hinges on their fully OSS (!) software stack, but they are getting things done, just had a new release: https://rocm.github.io
You don't call your PC a specific purpose computer just because all you do is use google chrome.
Potential customer: "What's an MPU?"
N: "Oh, it's the same thing we used to call a GPU"
P: "Why did you change the name?"
N: "Because it's a little bit weird to keep using the word 'graphics' for something that is being made more and more specifically for other things"
P (puzzled, and marginally less likely to make a purchase): "Oh right"
The new tesla volta super flip flop at 1.21 gigawatts blah blah blah.
Just saying.
Also Kudos to Nvidia for the buzzword/made up word creation for their products.
It's interesting the speed up isn't more pronounced between Volta and Pascal considering the Tensor cores on paper give you about 6x the MFlops. The price differential looks large.
From AnandTech: "By the numbers, Tesla V100 is slated to provide 15 TFLOPS of FP32 performance, 30 TFLOPS FP16, 7.5 TFLOPS FP64, and a whopping 120 TFLOPS of dedicated Tensor operations. With a peak clockspeed of 1455MHz, this marks a 42% increase in theoretical FLOPS for the CUDA cores at all size. Whereas coming from Pascal, for Tensor operations the gains will be closer to 6-12x, depending on the operation precision."