It is a very aggressive promotion, but I kind of wonder whether there's any chance it'll find a useful target audience. At least when we briefly looked at deploying various kinds of coprocessor solutions, paying the real cost for a development unit would not have been any kind of a blocker. It's just that actually deploying a Phi in production didn't seem to make any sense for us in the end (just like GPU and ASIC based options didn't either). I would imagine that a development unit would not have been an issue for anyone else who has seriously evaluated one of these. Seems like this kind of fire sale would only help in reaching people who need lots and lots of Phis, don't realize it yet, and are susceptible to buying a piece of kit just because it's almost free. (And admittedly very cool).
Also, my understanding is that this model is a 270W TDP board with passive cooling. What kind of a machine can you actually install those in?
You're meant to be running these in server rack cases that already have fans in the front. Most server CPU heatsink designs take this sort of form as well[1], as there's no point in putting a tiny fan on top of the heatsink when you've got delta fans at the front blasting air through what amounts to a sealed pipe. I've dealt with GPUs of a lower TDP in a cramped environment and the amount of airflow you need is completely insane, if you ran this without aggressive cooling for more than a couple of seconds it would just ignite.
I'm considering Phis for production workloads, and getting the test unit was never an issue. However, I also have some hobby projects that might work well on Phis. There's always the argument that personal projects are preparation for work stuff, but I'd be more comfortable running the former on my own hardware.
FWIW, I think I could fit two in the 4-way workstation I left in my parents' basement ;)
A full price board would require a procurement in many education/research organizations, but now you can just buy it through same channel as a spare laptop battery (ask your lab engineer or whatever).
Also, think China or India.
As for the cooling question, this is designed for rack servers that have a good front to back forced airflow. No doubt people will try various tricks at fan mounting to run in a desktop PC. See one recipe here: http://openwall.info/wiki/internal/xeon_phi
You'd be surprised at how much innovation comes bottom-up. Someone tries a neat, work-relevant technology for a pet weekend project, demos it, and the company adopts it.
There are places with heavyweight evaluation/adoption/etc. processes, but there are more agile ones as well. This kind of firesale does pretty well for that model.
Perhaps the insanely low price is because no one is buying the current design in the quantities they naively expected and they have to dump these ugly monstrosities for whatever price they can get before a new architecture is released.
I've said it before but I think it's worth repeating: general purpose CPU cores are not becoming much faster. Learning to make your software run well on multiple cores seems to be a very sound investment.
An insanely low price relative to it's original price. Peronally, I still find this a very reasonable price (double it would still be decent), but normally they are heavily overpriced.
It's hard to compare different devices of course, but in terms of pure FLOPS (and benchmarks) is roughly the same as an AMD r280x, which costs 240$ and allows you to play video-games in between your work.
There is definitely a market for these cards I think, niche but it exists, however the normal price of >$2k is the old-fashioned "industry tax" (c.f. Tesla cards) and not a reflection of the actual performance.
and allows you to play video-games in between your work.
That's what I was thinking too - if Intel just added the needed hardware for some video outputs, they could've produced a version which also works as a GPU, and might receive more interest.
Xeon Phi is the successor of Larrabee, a GPU project that failed because it couldn't reach the power/performance to compete with GPUs of the time (only 4 years ago)
Full fledged? They are basically just scaled down P54C (P5 architecture from 1993) cores that have upgraded functional units, and you have 57 to 61 one them. Flexible, maybe, but not as generally useful unless you are explicitly writing code for it.
A better buy would be an AMD APU, which for ~150-200 watts (compared to the 225-300 for the Phi, which requires a ~130W CPU to operate) can do roughly the same processing work with a little less onboard memory.
- It doesn't support CMOV, MMX, SSE, AVX, and all other ISA extensions the popped up after the original Pentium.
- It does not use standard System V x86-64 ABI.
- Most of its compute power is due to special vector instruction set, that is not compatible with any previous or future x86 ISA (including AVX-512).
- The only fully supported compiler is Intel Compiler (icc). And in my opinion, its code-generation quality for Xeon Phi is far worse than for other Intel architectures.
- The only supported assembler is GAS. No NASM, no YASM.
- Most debugging and profiling utilities (that run fine on normal x86 cores) are not supported on Xeon Phi. This includes valgrind, Address Sanitizer, Thread Sanitizer, and even memory profiling options in Intel compiler.
Given these limitations, how does it help that Xeon Phi is x86-based?
I'm a little confused about the value proposition here. This is about 60 CPUs at 1GHz with little memory and low IPC. Price, before promotion, is $2k, and after, $250.
I can get an FX-9590 for $250, which has 8 cores, but running at 4.7GHz, and higher IPC. In terms of raw compute speed, it seems like the FX-9590 will be at least the same speed. But:
* I can use standard programming tools.
* I only distribute computes 8-ways, not 60-ways. That's easier.
* Anything which is not easily made parallel is much faster. My experience is that there tends to be a lot of that.
* I have less work to transfer data back and forth. For big data, that actually takes a fair bit of time.
When would the Phi be faster? When would I want to use it?
In terms of raw compute speed, it seems like the FX-9590 will be at least the same speed.
They aren't really comparable. An FX-9590 at 4.7GHz is about 300 GFlops, but the the Xeon Phi 31S1P is slightly over 1 TFlop --- more than 3x as many floating point operations per second. The spec that probably shows the most difference is memory bandwidth. The FX-9590 with 1866 memory achieves 30GB/s. The Phi reaches 320GB/s, about 10 times as much. The presumption is that you might put 4 of these in a server, in addition to the standard processors.
The short answer would be that the Phi might be the best choice in cases where you need to perform billions of energy efficient floating point operations on a working set small enough to fit in the cards RAM, where the degree of branching is such that a "normal" GPU would be inappropriate, and where the problem justifies considerable programmer time for optimization. Financial and scientific models are the usual examples.
Ok, but if the normal price is 2000USD, then that leaves space for a lot of 9590s. You could get almost the same memory bandwidth and 3x as much processing power with 10 9590s. And remember that's floating point performance; the bulldozer architecture the 9590 is based on gives you about effectively 1.5x as many cores when doing integer operations.
So, ok, the Phi is useful at the current price, but at the 2000USD price other people are talking about in the thread it seems pretty useless.
Everyone in this thread keeps railing on its floating point perf, which is about on par with parts like nVidia's Tesla and similarly priced... but its integer performance is why people are really buying these cards up. All GPUs are floating point monsters but integer pansies because that's what their graphics workload heritage turned them into. Xeon Phi is a Computing Monster, with crazy Integer and Floating Point perf, because its heritage was "Let's try to make a graphics card out of CPUs" and not visa versa. It's similar to buying a whole stack of x86 chips just to get to the SSE/AVX units. Only, now you don't need a whole stack of computers, you need a handful of host computers and a stack of add-on cards.
And if you think $2000 is expensive for 60 x86 cores (~$35/core) when all you care about is the SSE/AVX unit and how many vector ops you can cram through it, you're definitely not Intel's target.
Thanks, you are right that I shouldn't have focussed on just floating point. I emphasized it because it better matched the frequently quoted "teraflop in a single card" measurement. As you say, the Phi is equally strong at vector integer operations as floating point, in both cases operating on 512-bit vectors.
But I'll quibble with the assertion that all GPU's are "integer pansies", although it's a fantastic phrase. Historically, NVidia GPU's are much stronger at floating point operations than integer because of their graphics origins, and NVidia's Tesla is definitely what Intel views as nearest to the Phi's target market.
But AMD/ATI cards have excellent vector integer performance, several times that of NVidia/Tesla, and much better per dollar than Phi until this recent price drop. This is why AMD cards were the best choice for bitcoin mining until the recent arrival of dedicated ASIC's: http://www.extremetech.com/computing/153467-amd-destroys-nvi...
> The short answer would be that the Phi might be the best choice in cases where you need to perform billions of energy efficient floating point operations on a working set small enough to fit in the cards RAM
I'm confused by this. You previously stated that memory bandwidth on the Phi is 10x as high as on the FX-9590. Based on this, I would say that the Phi would be better suited at jobs which require lots of data that doesn't fit in main memory.
I might be wrong, but I think his speeds are from the internal ram of the Phi and the FX-9590 to their processors, not the transfer speed from the hard drive to the cards ram. So if all of the data fits in the Phi's ram it can transfer that to the processors really fast, but if you have to transfer over the PCI you are limited to those speeds.
You are right. The largest Xeon Phi's have 16GB of GDDR5 memory on the card, and the 31S1P on sale has 8GB. The quoted speeds are for 16-channel transfer between the card's on board memory and the cores on the same card. Transfer from main memory back and forth to the card is over PCIe v2 x16 which has a theoretical max of 8 GB/s --- about 3% of the maximum transfer rate on data within the card.
So if your working data set fits on the card (less than 8/16 GB), or can be partitioned so that it fits on multiple cards, you can potentially get great performance from Phi. But if it's larger than this, especially if it is less than 1TB and would thus fit in the host's RAM, transfer from the host to the card will likely destroy any performance advantage.
It is my understanding that it sits in between a GPU and a CPU in terms of "ideal work". Like a GPU it favours SIMD work, but it is better at dealing with branches and a lot of random memory accesses[1] (it still favours linear work of course). It also uses a more familiar architecture, leading Intel to claim it is easier to master than GPU-programing, but I personally disagree as x64 is already much more complex than GPUs.
In terms of FLOPS the FX-9590 is still significantly slower. I don't know the exact numbers, but it will be hundreds of MegaFlops, whereas the Phi runs at ~1 TFlop. In part because the cores are basically low-end (in-order) atoms with modules bolted on for HPC (avx-512 instructions, instructions for exponentiation for example).
Data transfer to the card is a lot slower than reading from ram, correct, but on the other hand once it is on the card it is a lot faster (wiki says the fastest Phi has 350 Gb/s, Corsair claims a maximum of 70 Gb/s for DDR4)[2].
So generally, as long as you have a decent ratio of work/memory access and your work is parallel you could use a Phi to speed things a long. You would want to use one in the same situations as were you'd want to use a GPU, and the Phi would be preferable if your computations and memory access patterns are nor perfectly homogeneous (e.g. branching which can't be rewritten), as this kills performance on GPUs.
[1] I'm pretty sure Intel claimed this, and it makes sense when you think about it, but I can't seem to find a source: Could someone confirm this?
[2] Do note that it 350Gb/s shared between the cores, not 350 Gb/s each.
I'd be interested to hear about your experiences. By "more complex", do you mean the x64 instruction set itself, or the out-of-order superscalar processors that use it?
Basically the whole package. The instruction set is pretty big, but this is not much of an issue as you generally just need a subset for a given problem (I generally only tweak SSE/AVX code) and it has a lot of if's and but's when it comes to performance, the out-of-order execution also adds a degree of uncertainty, branch prediction and prefetching, a complex(ish) cache... There is a lot of things one can fiddle with in order to improve performance.
GPUs are much stricter in their design, which can lead to some headaches when having to completely change the way you formulate a problem, but once you're past that step, the simplicity of the architecture makes it a lot nicer to optimize in my opinion. (And my personal experience has been that the tools are cheaper (free) and more stable as well, but your mileage may vary).
The Xeon Phi uses a different computing architecture than either a CPU or a GPU so some intuitions about using it will be off. The Phi is essentially a modern barrel processor that uses the AMD64 ISA (it does not understand legacy x86 modes), a really nice vector ALU implementation, and memory bandwidth that is more like a GPU than a CPU. While it will run normal 64-bit software reasonably well with no special considerations, it will not be efficient without tweaking code design. CPU-targeted code attempts to optimize IPC in a single thread; barrel processors are designed to hide latency and can't drive IPC through single thread optimization.
The reason barrel processors are interesting is that they can be incredibly efficient with their clock cycles. Unlike either CPUs or GPUs, it is relatively easy to get sustained throughput that approaches the theoretical IPC of the silicon for diverse software. The Xeon Phi mentioned in the article has 114 ALUs; it is possible to ensure all of those ALUs are doing useful work every single clock cycle, unlike the much smaller number of ALUs in your CPU. CPUs and GPUs have higher theoretical throughput in some cases but various parts of their ALUs typically spend a significant part of their time idle.
Contrary to marketing, you do not want to program these like an ordinary CPU even though the cores are truly general purpose (unlike a GPU). Thread behavior is unlike CPUs or GPUs. Barrel processors cannot saturate a single core with a single thread! The Xeon Phi has 228 independent threads and you need to use them all the time.
The way barrel processors work is if the hardware supports N threads then each clock cycle you can saturate all the ALUs if some subset M of those threads are not stalled. The M-of-N ratio varies by barrel processor design but is typically 20-50% in my experience. Each clock cycle, a core selects an immediately runnable operation from the basket of threads it can see and executes it; as long as something is runnable in that basket, the core will do real work that clock cycle. Xeon Phi has a 50% M-of-N requirement, so you need a minimum of 114 threads that are not stalled every clock cycle to saturate the processor. The way you ensure that you hit the 114 threshold is to schedule all 228 hardware thread slots with useful work.
For programming, this changes the way you reason about locality and concurrency. If we assume that some percentage of threads can be safely stalled or blocked with no impact on throughput then it changes the way you design your algorithms and data structures. A little additional latency on a subset of threads won't hurt performance, especially if it increases task concurrency. On a CPU stalled threads are expensive, as it leads to idle cores or context switches. For architectures like Phi, you design your data structures and algorithms around relatively small, semi-independently work units so that there is always a large number of tasks that can be assigned to a thread even though this reduces locality. Below a certain threshold, thread concurrency is approximately free because the cores will schedule around stalls due to contention.
I like barrel processors quite a lot. Once you get used to the model, it is an easier architecture with which to achieve efficient massively threaded parallelism and throughput than either CPUs or GPUs. More importantly, they are hard to beat for efficiency for general purpose computing when software is designed for the architecture since so few clock cycles are wasted.
Coprocessors target audience has been for the most part HPC providers. If you're doing large numerical task or some huge amount of parallelism they can be incredibly useful. That being said there's a not insignificant amount of development time needed to properly use them.
Does anybody know how these sorts of devices are presented to an operating system? Is it controlled/programmed in a similar manner to a modern GPGPU card?
They run an embedded Linux operating system on the coprocessor, which in principle allows you to run programs on it, as if it were an ordinary albeit weird independent computer (see here https://software.intel.com/en-us/articles/building-a-native-...). That is you can ssh on the device, etc. There are also various tools sold by Intel all with three digit price tags (I believe), that allow you to program with parallel directives for example and which translate that to something that runs on a xeon phi. You can also use OpenCL, in which case it presents itself as one of several OpenCL devices you can use.
Whenever I improve the title, it gets reverted back to the original. Whenever I point to a secondary source, it gets replace with the primary. When I use the primary source with the original title, it gets no votes.
Looking at GFlops, much worse than GPUs from few years back (and we are in the ASICs era now). Don't know about scrypt, but since there are also ASICs there, I would imagine cost and power usage per hashrate would be not even comparable.
The Xeon Phi looks really awesome, and I'd love to be able to have that many cores at my disposal...however, as far as I can tell, its not at all like have 57 cores on your current computer. Whereas in my current software (Ruby, Haskell programs) I can just specify the number of threads to create to parallelize my software further, I'm not really sure how I'd use the Phi.
I seem to recall a case where someone used a GPU to handle index lookups or some other part of query processing for a database, on a GPU. I wonder if this could be used for that purpose.
Knight Landing, the second version, is supposed to be coming out in early 2015. If you want to get one be warned you might need a special motherboard as well if you were thinking of tossing it in a consumer box.
Won't be until the second half of 2015... We should have a better estimate come ISC in June, and my bet is on a release in November of 2015 to coincide with SC15.
Is there a cheap way to connect this to my Macbook pro? I do a lot of MD simulations, and this would be great to play around with when I don't want to use research funded computing time, but I feel like the connection to my laptop could end up being more expensive than the board itself.
You can probably find a Thunderbolt-connected PCIe enclosure, but the bandwidth from host memory to card memory will be lower than a direct connection. Drivers would be another issue.
I would be really tempted if I can find a computer capable of running it on the cheap (student). I think my current desktop may work, if I remove the video card and use the integrated graphics...
70 comments
[ 3.5 ms ] story [ 153 ms ] threadAlso, my understanding is that this model is a 270W TDP board with passive cooling. What kind of a machine can you actually install those in?
[1]: https://i.imgur.com/yh2oMeh.png
FWIW, I think I could fit two in the 4-way workstation I left in my parents' basement ;)
Also, think China or India.
As for the cooling question, this is designed for rack servers that have a good front to back forced airflow. No doubt people will try various tricks at fan mounting to run in a desktop PC. See one recipe here: http://openwall.info/wiki/internal/xeon_phi
There are places with heavyweight evaluation/adoption/etc. processes, but there are more agile ones as well. This kind of firesale does pretty well for that model.
:(
[1]: https://software.intel.com/en-us/mic-developer#pid-22612-186...
- Black-Scholes Valuation Computing
- Weather simulation
fairly specialist number crunching I guess
[update]
I see the 'worlds fastest computer', Tianhe-2 uses 48,000 Xeon Phis. Wonder if they got a bulk discount?
https://en.wikipedia.org/wiki/Tianhe-2#Applications
It's hard to compare different devices of course, but in terms of pure FLOPS (and benchmarks) is roughly the same as an AMD r280x, which costs 240$ and allows you to play video-games in between your work.
There is definitely a market for these cards I think, niche but it exists, however the normal price of >$2k is the old-fashioned "industry tax" (c.f. Tesla cards) and not a reflection of the actual performance.
That's what I was thinking too - if Intel just added the needed hardware for some video outputs, they could've produced a version which also works as a GPU, and might receive more interest.
A better buy would be an AMD APU, which for ~150-200 watts (compared to the 225-300 for the Phi, which requires a ~130W CPU to operate) can do roughly the same processing work with a little less onboard memory.
- It doesn't support CMOV, MMX, SSE, AVX, and all other ISA extensions the popped up after the original Pentium.
- It does not use standard System V x86-64 ABI.
- Most of its compute power is due to special vector instruction set, that is not compatible with any previous or future x86 ISA (including AVX-512).
- The only fully supported compiler is Intel Compiler (icc). And in my opinion, its code-generation quality for Xeon Phi is far worse than for other Intel architectures.
- The only supported assembler is GAS. No NASM, no YASM.
- Most debugging and profiling utilities (that run fine on normal x86 cores) are not supported on Xeon Phi. This includes valgrind, Address Sanitizer, Thread Sanitizer, and even memory profiling options in Intel compiler.
Given these limitations, how does it help that Xeon Phi is x86-based?
I can get an FX-9590 for $250, which has 8 cores, but running at 4.7GHz, and higher IPC. In terms of raw compute speed, it seems like the FX-9590 will be at least the same speed. But:
* I can use standard programming tools.
* I only distribute computes 8-ways, not 60-ways. That's easier.
* Anything which is not easily made parallel is much faster. My experience is that there tends to be a lot of that.
* I have less work to transfer data back and forth. For big data, that actually takes a fair bit of time.
When would the Phi be faster? When would I want to use it?
They aren't really comparable. An FX-9590 at 4.7GHz is about 300 GFlops, but the the Xeon Phi 31S1P is slightly over 1 TFlop --- more than 3x as many floating point operations per second. The spec that probably shows the most difference is memory bandwidth. The FX-9590 with 1866 memory achieves 30GB/s. The Phi reaches 320GB/s, about 10 times as much. The presumption is that you might put 4 of these in a server, in addition to the standard processors.
The price is actually quite a bit better, too, as some vendors are offering the Phi at $125 each in quantities of 10: http://www.colfax-intl.com/nd/xeonphi/31s1p-promo.aspx
This article offers some good information about cases where the Phi is a good (or bad) choice: https://software.intel.com/sites/default/files/article/33016...
The short answer would be that the Phi might be the best choice in cases where you need to perform billions of energy efficient floating point operations on a working set small enough to fit in the cards RAM, where the degree of branching is such that a "normal" GPU would be inappropriate, and where the problem justifies considerable programmer time for optimization. Financial and scientific models are the usual examples.
So, ok, the Phi is useful at the current price, but at the 2000USD price other people are talking about in the thread it seems pretty useless.
And if you think $2000 is expensive for 60 x86 cores (~$35/core) when all you care about is the SSE/AVX unit and how many vector ops you can cram through it, you're definitely not Intel's target.
But I'll quibble with the assertion that all GPU's are "integer pansies", although it's a fantastic phrase. Historically, NVidia GPU's are much stronger at floating point operations than integer because of their graphics origins, and NVidia's Tesla is definitely what Intel views as nearest to the Phi's target market.
But AMD/ATI cards have excellent vector integer performance, several times that of NVidia/Tesla, and much better per dollar than Phi until this recent price drop. This is why AMD cards were the best choice for bitcoin mining until the recent arrival of dedicated ASIC's: http://www.extremetech.com/computing/153467-amd-destroys-nvi...
> It doesn't support CMOV, MMX, SSE, AVX, and all other ISA extensions the popped up after the original Pentium.
I'm not saying it's not useful, it's just not quite as useful as having 60 cores that can just run MMX/SSE/AVX.
> It doesn't support CMOV, MMX, SSE, AVX, and all other ISA extensions the popped up after the original Pentium.
I'm not saying it's not useful, it's just not quite as useful as having 60 cores that can just run MMX/SSE/AVX.
I'm confused by this. You previously stated that memory bandwidth on the Phi is 10x as high as on the FX-9590. Based on this, I would say that the Phi would be better suited at jobs which require lots of data that doesn't fit in main memory.
So if your working data set fits on the card (less than 8/16 GB), or can be partitioned so that it fits on multiple cards, you can potentially get great performance from Phi. But if it's larger than this, especially if it is less than 1TB and would thus fit in the host's RAM, transfer from the host to the card will likely destroy any performance advantage.
In terms of FLOPS the FX-9590 is still significantly slower. I don't know the exact numbers, but it will be hundreds of MegaFlops, whereas the Phi runs at ~1 TFlop. In part because the cores are basically low-end (in-order) atoms with modules bolted on for HPC (avx-512 instructions, instructions for exponentiation for example).
Data transfer to the card is a lot slower than reading from ram, correct, but on the other hand once it is on the card it is a lot faster (wiki says the fastest Phi has 350 Gb/s, Corsair claims a maximum of 70 Gb/s for DDR4)[2].
So generally, as long as you have a decent ratio of work/memory access and your work is parallel you could use a Phi to speed things a long. You would want to use one in the same situations as were you'd want to use a GPU, and the Phi would be preferable if your computations and memory access patterns are nor perfectly homogeneous (e.g. branching which can't be rewritten), as this kills performance on GPUs.
[1] I'm pretty sure Intel claimed this, and it makes sense when you think about it, but I can't seem to find a source: Could someone confirm this? [2] Do note that it 350Gb/s shared between the cores, not 350 Gb/s each.
I'd be interested to hear about your experiences. By "more complex", do you mean the x64 instruction set itself, or the out-of-order superscalar processors that use it?
GPUs are much stricter in their design, which can lead to some headaches when having to completely change the way you formulate a problem, but once you're past that step, the simplicity of the architecture makes it a lot nicer to optimize in my opinion. (And my personal experience has been that the tools are cheaper (free) and more stable as well, but your mileage may vary).
The reason barrel processors are interesting is that they can be incredibly efficient with their clock cycles. Unlike either CPUs or GPUs, it is relatively easy to get sustained throughput that approaches the theoretical IPC of the silicon for diverse software. The Xeon Phi mentioned in the article has 114 ALUs; it is possible to ensure all of those ALUs are doing useful work every single clock cycle, unlike the much smaller number of ALUs in your CPU. CPUs and GPUs have higher theoretical throughput in some cases but various parts of their ALUs typically spend a significant part of their time idle.
Contrary to marketing, you do not want to program these like an ordinary CPU even though the cores are truly general purpose (unlike a GPU). Thread behavior is unlike CPUs or GPUs. Barrel processors cannot saturate a single core with a single thread! The Xeon Phi has 228 independent threads and you need to use them all the time.
The way barrel processors work is if the hardware supports N threads then each clock cycle you can saturate all the ALUs if some subset M of those threads are not stalled. The M-of-N ratio varies by barrel processor design but is typically 20-50% in my experience. Each clock cycle, a core selects an immediately runnable operation from the basket of threads it can see and executes it; as long as something is runnable in that basket, the core will do real work that clock cycle. Xeon Phi has a 50% M-of-N requirement, so you need a minimum of 114 threads that are not stalled every clock cycle to saturate the processor. The way you ensure that you hit the 114 threshold is to schedule all 228 hardware thread slots with useful work.
For programming, this changes the way you reason about locality and concurrency. If we assume that some percentage of threads can be safely stalled or blocked with no impact on throughput then it changes the way you design your algorithms and data structures. A little additional latency on a subset of threads won't hurt performance, especially if it increases task concurrency. On a CPU stalled threads are expensive, as it leads to idle cores or context switches. For architectures like Phi, you design your data structures and algorithms around relatively small, semi-independently work units so that there is always a large number of tasks that can be assigned to a thread even though this reduces locality. Below a certain threshold, thread concurrency is approximately free because the cores will schedule around stalls due to contention.
I like barrel processors quite a lot. Once you get used to the model, it is an easier architecture with which to achieve efficient massively threaded parallelism and throughput than either CPUs or GPUs. More importantly, they are hard to beat for efficiency for general purpose computing when software is designed for the architecture since so few clock cycles are wasted.
The entire thing (with memory and disk) took up 1600 sq feet of floor space.
Here we are 17 years later, and we can get the same processing power in a pci card.
http://www.pugetsystems.com/blog/2013/08/06/Will-your-mother...
Here is a link to Intel's promotion page it has participating vendors in various locations:
https://software.intel.com/en-us/articles/special-promotion-...