Shit, I did not realize GPUs packed such a tremendous performance advantage over CPUs (FLOPs wise). They seem to be several orders of magnitude ahead--that is positively mindblowing!
They compare instructions per second with nerve impulses per second. That's a pretty useless comparison. I'm sure that all GPUs and CPUs could simulate a human brain more quickly than a human brain could simulate all CPUs and GPUs.
In the Whole Brain Emulation Roadmap there's a table of published guesses at computing requirements: most were in the range 10^14..10^18 FLOPS. (Plus a few outliers from people who think the brain does molecular-scale or quantum computing in microtubules.) The 500th biggest computer in the latest Top500 does around 3E13 FLOPS, while the top does 3E15 -- using GPUs, incidentally. Your link's on the high end of the range, though they're all kind of guessing.
There are many architectural differences between CPUs and GPUs that make it difficult to draw the kinds of conclusions suggested by these graphs.
Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.
Very true. It is for the same reason that the article shows GPU progress outpacing CPU's in regards to FLOPS. The limited scope of a GPU's functionality allows most of the processor's silicon to be arithmetic units. CPU's, on the other hand, have huge sections of their real estate dedicated to handling conditional logic (something most GPU's fail miserably at), legacy instruction handling, virtualization, etc.
If someone figured out how to write regular apps on a GPU (which would amount to parallelizing the app), it could change everything.
1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way.
2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow?
3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).
It can't really be done trivially. A GPU is a massively parallel (hundreds of cores), very limited type of CPU, specifically made to execute vector operations.
I agree, and I can't wait for someone (or, more likely, some group) to figure out how to parallelise arbitrary programs across GPUs and CPUs in a way that is transparent to the programmer - or as nearly so as possible.
I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-)
No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
This is what parallelizing Fortran compilers do. The space of programs that can be automatically parallelized is all computable programs (see http://en.wikipedia.org/wiki/Church–Rosser_theorem). The space of programs that can be efficiently automatically parallelized is very small.
Many problems aren't parallelizable for reasonable definitions of the word (i.e. polylogarithmic depth and polynomial work: Nick's class) unless there are some major breakthroughs in complexity theory.
As a class of problems, probably (although it hasn't been proven either way), but in terms of programs it comes down to the vagaries of data structures and algorithms - if the dataflow in your reduction graph makes it look like a linked list, there's only one way to reduce it.
The good thing is that functional programmers will barely notice that they are using the new data structures. (It's still just filters and maps. Just your fold-functions will take slightly different arguments.)
How are they going to get there? They could very easily pump up the shader count to get something that is "100x" performance, while not actually being 100x as powerful for general purpose operations. The graph looks shady to say the least.
Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die).
Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel).
GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize.
And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations.
Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth.
From the summary : While there is a definite speedup from running on 1 cpu only to running on 1 cpu + 1 gpu, the multi-threaded versions generally have better overall performance. It is likely that once the CUDA versions are multi-threaded, running on the GPU nodes will confer additional advantages, but at present the codes are not yet mature.
26 comments
[ 4.0 ms ] story [ 73.8 ms ] threadAnd yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?
http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3...
http://www.top500.org/list/2010/11/100
Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.
1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way.
2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow?
3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).
I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-)
No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
Guy Steele gave a really good talk about this at ICFP 2009: http://labs.oracle.com/projects/plrg/Publications/ICFPAugust...
It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chip...
Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die).
Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel).
GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize.
And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations.
Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth.
Fun times!
http://biowulf.nih.gov/apps/bioinf-gpu.html
From the summary : While there is a definite speedup from running on 1 cpu only to running on 1 cpu + 1 gpu, the multi-threaded versions generally have better overall performance. It is likely that once the CUDA versions are multi-threaded, running on the GPU nodes will confer additional advantages, but at present the codes are not yet mature.