What's worse is that the U.S prevented China from buying Intel chips to build their supercomputer with[1]. It was a huge order and probably had something to do with the recent layoffs.
The worst is yet to come. China's home grown chips may now spread out from China to undercut Intel at the end high cloud market if Linux-based software is easy to port to it. Intel's CPUs are way over priced and ripe for disruption if someone can get the required massive upfront investment to create a real competitor, and it looks like China is doing this. Repeatedly topping the supercomputer list is perfect marketing for introducing a new high performance CPU to the world.
I wonder what the process node size of SW26010 is and if the chips were actually manufactured in China or if they were manufactured by TSMC/Taiwan.
I wasn't aware that China had competitive fabrication plants for processors. If they don't and they built these at say 40 or 45nm then wouldn't this design performance be even more impressive?
CPU performance is not that important when building a supercomputer it's more budget * performance per node / price per node. Performance per node is useful as most workloads only scale so far.
Yes, but that is exactly what people are generally looking for in cloud machines. But I guess we need a benchmark that is oriented to web application development such as serving N nginx/go/nodejs requests per second, or running a MySQL/Postgres/MondoDB server at a certain load.
A former NSA and CIA chief, wrote a post about stuff like this, before the US banned US chip makers from selling to China:
> By the time I became NSA director in the late 1990s, however, the calculation was no longer that simple. We still wanted an MTOPS advantage, of course, but we were fast realizing that our preferred limits were undermining the global competitiveness of the U.S. computer industry — the very industry on which we relied for our success. It was becoming clear that the overall health of that industry was more important than any MTOPS advantage against a specific target country. We still insisted on limits with regard to places such as Cuba and North Korea, but we became far more forgiving elsewhere.
This, of course, had a powerful, positive commercial impact, but the NSA didn’t flip its position for commercial reasons. We did it for security reasons. On balance, this change made us stronger, not weaker, over the long haul, since retarding exports would inevitably retard the technological progress that was both our economic and our security lifeblood.
That early lesson has caused me to continue to challenge arguments that technological protectionism furthers national security. It might, but then again, it could have the opposite effect if it freezes development, alienates allies, feeds distrust or invites the creation of similar barriers abroad. I would recommend these broader considerations to those in the U.S. security enterprise with responsibility for evaluating these trade-offs today.
The whole post is worth reading. The same logic could be applied to US trying to put backdoors in its products for a specific national security goal, which will ultimately end up undermining US technological supremacy and national security as well. And yet the current CIA director just implied that US would be fine with backdoored products, because "where are people going to get their encryption from? The foreigners? Ha!"
This sort of arrogance, which is the same type of arrogance that ended up banning chip sales to China last year, is what will make the US lose out in the long term.
Thanks for that article by Hayden. Wow. It echoed much of what Schneier et al kept saying to him. Shows that they're more honest, receptive, or both when they finally leave the job that forces them to BS for SIGINT gains. ;)
I've hearing this for over three decades: "Country X is on the brinking of beating US general computing lead." I must have shuddered in over 60 pairs of boots since then.
Except it happened and you didn't even notice. Much computer electronics is designed and made in China. And today this: "China Tops U.S. in Supercomputers".
> In a notice published online the US Department of Commerce said it refused Intel's application to export the chips for Tianhe-2 and three other Chinese supercomputers because the machines were being used for "nuclear explosive activities".
Oh no, they're using their supercomputers to do nuclear research? That's horrible. Who would do such a thing?
The given reason is all bullshit, and they must think we're all morons if they expect us to buy it. If anything it's retaliation for some IP theft or some silent trade war going on between China and the US. It's certainly not because the US gov is "shocked" that China uses its supercomputers to do nuclear research.
If you want to sell technology products in the China market, you need to transfer technology to China, such as setting up a R&D centre. Everyone answers to sales.
Then comes Machiavellian management strategies that demonstrate large companies are not idiots, and withdrawal of such scheme 2-5 years later, while still holding sales lines in China.
Source: Have worked on both sides of the technology transfer equation.
Simulating nuclear explosions in software has been a major response to treaties that forbid nuclear tests. As I think you were ironically pointing out, it's common now.
It might be hard for the U.S. government to effectively prevent the Chinese government from doing these simulations, but it's plausible to me that the U.S. government would still want to see the Chinese government hindered in nuclear weapons simulations.
Naturally, the people who are modding you down are conspicuously silent on why it's necessary or even desirable to make it harder for China to simulate nuclear weapons.
I have a question. 125.4 Pflop/s ... that would be about 23k nvidia GPUs (granted, 1080s). They claim they'll be able to do that with 10k by the end of the year, beginning of next year with server-class GPUs.
So that Chinese number seems awfully low to me. I would expect that the number Amazon has to be higher, for instance. Same for Microsoft and Google. Therefore I'd be amazed if the DoD, NSA and even the DoE wouldn't have more capacity available.
But if you have an application that can be distributed transparently to thousands of GPUs, the difference between one and several systems is not very relevant.
It is, because some problems can be distributed to 1000s of GPUs and done in parallel, and some can't, because the answer to one calculation depends on the answer to another calculation. You could combine the computing power of Azure, AWS and Google, and be pretty disappointed because of all the waiting time due to latency from one data center to another That's when the system's architecture - software and hardware - becomes important.
Many problems are very latency tolerant. Unless you have an algorithm which is truly latency intolerant, I argue you are best served not investing in low-latency interconnect because it costs so much.
When I ran Exacycle, we distributed protein folding, protein design, drug discovery, telescope design, and other problems globally. We never had an issue with latency, because these problems all partition really well. People who claim supercomputers are "necessary" for these problems typically construct problems that are well-matched to supercomputers (for example, running molecular dynamics on huge proteins) but they tend not to have very high scientific value.
In my experience, partitioning to minimize communication has always increased my total scientific throughput, while programming to supercomputers has always reduced it.
Your arithmetic is in the ballpark -- the article says this machine has 40k processors. But, firstly, these are double precision numbers, I believe that lowers nvidia performance substantially. Furthermore, scaling is not as trivial as putting them in the same room -- you need massive networking, cooling and power supply. Also, while the LAPACK benchmark can work pretty well on GPUs, the stuff that the computers actually is used for might not.
In practice you have many more challenging question that calculate how many GPUs would do the job. For starting how would you connect the GPUs? And so on.
Please excuse my ignorance in this area, I was just wondering - why does it matter so much who has the biggest and baddest supercomputer? Shouldn't that which we process on them matter more?
A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2. That kind of makes a supercomputer look like a pile of hardware, no matter how well it's organized or engraved.
Its essentially a matter of realistic benchmark acceptance. Right now machines are compared by how well they run this benchmark called LINPACK, which has been criticized for being non-representative of real science codes. As mentioned here [1], China's new system only achieved 0.3% of its peak flops on a slightly more realistic benchmark, HPCG.
While Sunway TaihuLight has weak HPCG performance, 0.3% number should be understood in context. Tianhe-2 (#2 system) scores 1.1%, and Titan (#3, US #1 system) scores 1.2% for HPCG. So Sunway TaihuLight is 3~4x worse for HPCG compared to other top systems.
Note that K computer (#5 system, Japan) scores 4.9% for HPCG. So Tianhe-2 and Titan are again 4x worse for HPCG compared to systems which score best for HPCG.
I don't think it's as simple as LINPACK bad, HPCG good. LINPACK is representative for some workloads, when compute dominates. HPCG aims to balance compute and memory. There is Graph500, if memory dominates for your workload.
>> rumors that China didn't even know what to process on their Tianhe-2.
Kind of reminds me of when we heard a rumor than a rival ISP had bought a handful of Sun servers (in the year 2000). We thought they had tends of thousands of customers, figured they would be launching DSL, figured that had thousands of web hosting customers...
Turns out they got some weird grant so they spent the cash on those, otherwise they would have lost the money. They went out of business a year later and we kept chugging along with our Pentium III servers.
This happened a lot with Sun around that time; I bought 10 E450s for fun from one of the hosters who did exactly that and since then went bankrupt. Great, unbreakable machines.
Bubble dot-com companies rich with VC cash did the same thing. Built massively overprovisioned data centers with expensive Sun servers (because Sun == Internet at that time) and never actually had the traffic to justify even a fraction of it.
> A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2. That kind of makes a supercomputer look like a pile of hardware, no matter how well it's organized or engraved.
There are enough high quality universites in China to make use of it. I can not imagine this is true. There may have not been one big algorithm that it was built for, but university researchers always have stuff to compute.
Well for one there's being able to control 51% of bitcoin[1] which would let China "prevent transactions of their choosing from gaining any confirmations, thus making them invalid, potentially preventing people from sending Bitcoins between addresses. They could also reverse transactions they send during the time they are in control (allowing double spend transactions), and they could potentially prevent other miners from finding any blocks for a short period of time."
Some other comment I read about this calculated that this supercomputer has about 0.1% or 0.01% of an impact on the total bitcoin calculating power, so effectively negligible. ASICs would work better.
>Please excuse my ignorance in this area, I was just wondering - why does it matter so much who has the biggest and baddest supercomputer? Shouldn't that which we process on them matter more?
Yes, until one starts to lose on most such trivial metrics, and then they have a more substantial problem than losing on trivial metrics...
Those things can be long term signs about where the future is going.
>A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2.
Yeah, those yellow people are so ignorant -- it takes white folks to show them what to process in their supercomputers. Sorry, but those rumours sounds openly racist.
You'd be surprised that people do buy supercomputers without a real idea (or only a rather vague general idea) of what they're going to do with it. And by people I mean universities. Not unheard of for a university to buy a supercomputer for essentially bragging rights, and then for it to sit underutilized.
I would think that getting simulation time on a top ranked super computer would be something that you would schedule months, if not years in advance - hard to imagine that having access to a multi-petaflop wouldn't be highly desirable (and useful) for many, many researchers, let alone all the graduate students who spend months of lower-tier computers churning out simulations...
I'd be interested to hear from someone who knows/has experience, whether any of the top 100 super computers ever have idle windows.
I worked on a system that was at one time #3 on the list. There were no substantial idle periods. If the machine was out of maintenance, jobs were being run on it. Depending on the internal structure of the machine, some parts of it might be idle as the scheduler drained other parts to allow space for larger jobs to start, but I wouldn't call it idle.
Having done exclusively HPC for almost 16 years now, I can state that there is one and only one immutable law of HPC: grad students will find a way to squeeze out every second of available CPU time on a large system. The amount of effort going into taking advantage of those empty spots in the clusters where smaller jobs can run quickly while the scheduler drains resources for a large job (called backfilling) is tremendous.
The thing that sticks with me is what a fantastically complicated problem HPC job scheduling is. I've seen dozens of fresh-faced undergrads or first year grad students come in with a full head of steam and decide that they're going to "solve" HPC job scheduling, but every single one has gotten bogged down in the minutiae and the muck, and I've never seen software that "gets it right". Every scheduler sucks in its own unique way.
Whether all of those jobs need an old-fashioned supercomputer is certainly up for debate, but they certainly get used, at least in my experience.
interesting, do you feel that the lack of idle periods was because it was being used for tasks that truly demanded and could not be performed without that compute power?
Or was it a combination of that with "well its there, and we want to do this thing that would happen quicker on the big machine" - IE not necessarily enabling new research, just adding a nice speedup to existing work?
You don't generally build a large supercomputer to simply run a task faster than on a smaller cluster. The goal is to enable more realistic simulations by solving problems using a greater number of unknowns, increasing the fidelity of output.
So no, these machines are often used for problems which really do utilize the large-scale nature of the computers.
I agree with tanderson92. Whether the actual science from all the simulations is "worthwhile" is for people much smarter than me, but to get time on a "leadership" class machine, you have to prove in your application that a) it will run properly on the hardware; and b) that it's science you wouldn't get done on smaller machines.
So lots of the "big data" type problems with lots of uncoordinated jobs aren't going to take advantage of the primary component of these big computers, the node interconnect. The computers really shine with big fluid dynamics simulations where there's lots of chatter between different components of the simulation to communicate boundary conditions and the like.
> A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2.
That applies to most of supercomputers everywhere. "If you build it, they will come".
Besides the knowledge attained in building supercomputers ( which all major nations should know how to do ) and processor technology, the added benefits and new uses of these machines will come later.
The internet was created before people really knew what to do with it. The same thing with computers. The same thing with everything else.
You would be correct to question if there is a first-order advantage to having the most powerful supercomputer. But it is a national strategic benchmark, like precision manufacturing, certain kinds of biotechnology, rocketry, etc.
What matters most might be that China is maturing CPU technology that isn't nearly as congenial for US TLAs to hack in to. It might not matter so much for this supercomputer, but if Chinese suppliers put price ahead of things like opaque management processors, it might lead to tipping the balance away from easily exploitable endpoints.
US is routinely enbarassed by a more accurate European weather forecast computation. Hurricane Sandy prediction was the glaring example. Part of the Euro prowess is due to better physics. But much of it to a finer model they can compute in real time of a larger computer. Everytime you double the resolution, the computing cost increases 16 times in this 4D computation.
You are implying that the accuracy is mainly due to greater compute. This is simply not true. If the US model could be improved by throwing money for compute at it, that would already have been done.
So what is the cost of these CPUs? Could then be used in desktops or more importantly various cloud services (AWS, Google, Microsoft)? What OS do these run?
Is there plans to commercialize these chips? We need accessible low cost ultra-high core count chips widely available. I feel that Intel and AMD have been under performing in this area (growing core count) for the last decade. And Intel has kept the price of CPUs exceptionally high for the last decade as well.
If low cost ultra-high performance chips from China under cut Intel's excessively priced Xeons (+$3K per CPU), it really could change things. If these start to spread, I wouldn't be surprised if Intel tried to prevent the spread of these CPUs outside of China via trade barriers.
Ultra-high performance and low cost don't belong in the same sentence. Even with intel if you want bang for buck you're better off buying 12 core CPUs than the 18 core ones.
Whilst touched on in the linked article but not mentioned is that the codes for the Gordon Bell runs are in the 30-40 petaflop range of sustained application performance. That's not too shabby.
According to TOP500 author Jack Dongarra, three scientific simulation codes run
on TaihuLight have been chosen as Gordon Bell Prize finalists, two of which have
managed to reach a sustained performance of 30 to 40 petaflops. The award is
bestowed each year on the most noteworthy HPC application, based on “peak
performance or special achievements in scalability and time-to-solution on
important science and engineering problems.”
Here's something that's always bothered me. In the HPC and scientific computing world, it seems that the word 'program' is never used to refer to what is being run on the computers, rather it is always 'codes' (note plural) that are executed. Does anyone know when or why this split of terminology occurred?
The truth is, US Super-Computer spending is hard to justify and most of the owners of these in various national institutes are shopping for customers to justify their funding, although their set-up is not a good fit for many types of computing.
Every dollar spent on building these computers is a dollar NOT spent in the budget of a research-driven grant. Give these dollars to physicists, mathematicians, genomics scientists, chemists, climate people etc. This way the demand in computing will dictate the type of systems built.
p.s. I've attended the event in person that's listed in the article hosted by the OSTP, which is a part of the White House (National Strategic Computing Initiative). My conclusion was most of these computers aren't needed and a misdirection of funds to the tune of billions of dollars that computing / scientific computing could really benefit from in other ways.
We don't need to communicate between the nodes; We don't need MPI. We need distributed databases and we might need Spark. We need machine learning. We need servers. We would like availability and reliability.
We get full batch nodes that are relatively anemic in memory for our workload (4GB/core) for a maximum of 24 hours at a time.
Basically it's hard to use a Cray node as a stand-alone server. Yes it runs linux, but it has no storage, no external network. There is the so-called cluster compatibility mode but bottom line everything is a scheduled batch job in a very walled garden.
All the stranger that they sold their interconnect technology, arguably their "secret sauce" to Intel. Without the interconnect, they are just a bunch of (very high quality) diskless x86 systems.
Huh. At the core of almost every problem I've ever encountered in my 20+ year career programming that I would consider "interesting" is the need to allocate work across many compute nodes that have some amount of shared state. You could just as well take A* as a baseline for these types of problems. Your typical cloud computing infra/design is not a good match for these.
Well I guess computing in astrophysics and HEP isn't that interesting then :)
It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state.
There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core).
So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.
Well this is where you talk to the network admins at your university and get them in on a top secret program, code name: "It Was China", to hack into all the detectable ps4's on campus, call all those students to a hall for a random lottery social for a couple of hours while you commandeer there equipment, for science of course.
Or you can send get the network admins to allow you to send them a email, but the first suggestion sounds more fun :P
1) I have a rail system that I want to optimize train movement through the topology. My search space is the number of running locomotives to the power of the number of switches in the system. On a continental rail system, that's a number with many, many, many zeros after it. To optimize this, I have many cores chewing through the space, but to know where to search they must share a visited node cache otherwise they will search each others space and waste effort.
2) I have a system that predicts part failures by collecting sensor data of the machine in a time series. I must look for patterns of change in these sensor readings and compare that to patterns that I know about, often drilling through data over many weeks looking for feature alignment in the sensor matrices. Same problem: many cores, some amount of space that is known vs. unknown that must be shared, but as a result is a contentious resource.
In many cases, problems partition nicely across machines without communication. Take molecular dynamics as an example of an "interesting" problem. One approach is MPI-based parallelism: partition the particles or other entities in the system across specific processors, compute all the individual forces, sum them up, and send the results to the processors that need it. This allows you a number of advantages: you can work with larger systems (ones that wouldn't fit in RAM on a single machine), you can speed up the total throughput, etc). But, it comes at a huge cost to developer productivity, and the machines have to have very low latency, high through primitives, so that all the processors are evenly balanced (Amdahl's law places upper bound on total throughput).
On the other hand, you can just run N independent simulations, and pool the statistics. In many cases, this is as good as if not better than running one long MD simulation. Here, all the processors share a common input file, but it's a single file, it only has to be communicated once. Any results from the processors are compacted, and sent to a large file server, which in itself is designed with partitioning since all the output files are independent. This approach won't let you work with large systems (unless your individual servers have more RAM), but it scales linearly to far more processors than tightly coupled parallelism, for far cheaper, and often, the scientific results are as good or better.
This shows up in many cases- in my 20+ year career programming, nearly all the problems I saw people run on supercomputers eventually could be run on CPUs with easier code and better performance. There are some irreducible codes for which only supercomputers are capable of meeting the required specs, and those are the only codes which should run on supercomputers. Otherwise you're pissing money on interconnect and hard to program environments).
In short: if you can convert your supercomputer job into a cloud cluster, you should do so, because it will save time and money. If you can't, then you should use a supercomputer.
Note also that cloud interconnects have gotten significantly better over the years- for me, 100Gbit throughput and latency (as long as the cluster has the total parallel bisection bandwidth) is fine for me and most other people.
> most of the owners of these in various national institutes are shopping for customers to justify their funding
What's the evidence for this claim? AFAIK the DoE INCITE awards -- which is how you get allocated time on the big supercomputers -- are quite competitive.
Also the supercomputers don't cost billions. Aurora, for example only costs $200 million, and if it lasts as long as its predecessor Mira did, that is $200 million spread over 6 years, which is a pretty small number.
> My conclusion was most of these computers aren't needed
I don't disagree, but I wonder what is behind the investment in China. They are putting a significant chunk of resources on this and I'd like to understand why.
Well, there's the PR value, both inside and outside the country. Maybe they're trying to convince investors that China's a good place to go if you want to fabricate computer chips.
Forgive my ignorance, but what's the point of a "super computer" in a world where we can have an application access thousands of machines around the world at once with a button click to compute something? I could see maybe cost savings, but considering how much it costs to build the machine and that I doubt it's used to 100% capacity at all times. Is it a security thing, or just a pride thing? What's the advantage over just farming it out to a cluster in the cloud?
Traditionally, supercomputers and mainframes have way higher inter-node IO performance like bandwidth than anything you can spin up on EC2. IO is extremely important because it's very rare to have a true shared-nothing algorithm, which means that Amdahl's Law will bite you and make the little bit of IO and coordination a dominant factor. For example, in deep learning, an awful lot of the work is simply moving around and updating data, which has been a bottleneck for attempts to train NNs on multiple GPUs.
The Chinese government has been committing industrial theft on a large scale for years. This is well-documented but somehow China has gotten away with it.
100 comments
[ 3.7 ms ] story [ 136 ms ] thread[1] http://nextbigfuture.com/2016/05/us-supercomputer-chip-ban-d...
http://www.nextplatform.com/2016/06/20/look-inside-chinas-ch...
Summary, it's a RISC architecture that resembles an Alpha but it's not a straight-up copy.
I wasn't aware that China had competitive fabrication plants for processors. If they don't and they built these at say 40 or 45nm then wouldn't this design performance be even more impressive?
> By the time I became NSA director in the late 1990s, however, the calculation was no longer that simple. We still wanted an MTOPS advantage, of course, but we were fast realizing that our preferred limits were undermining the global competitiveness of the U.S. computer industry — the very industry on which we relied for our success. It was becoming clear that the overall health of that industry was more important than any MTOPS advantage against a specific target country. We still insisted on limits with regard to places such as Cuba and North Korea, but we became far more forgiving elsewhere.
This, of course, had a powerful, positive commercial impact, but the NSA didn’t flip its position for commercial reasons. We did it for security reasons. On balance, this change made us stronger, not weaker, over the long haul, since retarding exports would inevitably retard the technological progress that was both our economic and our security lifeblood.
That early lesson has caused me to continue to challenge arguments that technological protectionism furthers national security. It might, but then again, it could have the opposite effect if it freezes development, alienates allies, feeds distrust or invites the creation of similar barriers abroad. I would recommend these broader considerations to those in the U.S. security enterprise with responsibility for evaluating these trade-offs today.
https://www.washingtonpost.com/opinions/dont-let-america-be-...
The whole post is worth reading. The same logic could be applied to US trying to put backdoors in its products for a specific national security goal, which will ultimately end up undermining US technological supremacy and national security as well. And yet the current CIA director just implied that US would be fine with backdoored products, because "where are people going to get their encryption from? The foreigners? Ha!"
This sort of arrogance, which is the same type of arrogance that ended up banning chip sales to China last year, is what will make the US lose out in the long term.
https://www.techdirt.com/articles/20160618/08022234741/cia-d...
What more evidence do you want?
http://nextbigfuture.com/2015/04/us-will-make-180-petaflop-s...
The given reason is all bullshit, and they must think we're all morons if they expect us to buy it. If anything it's retaliation for some IP theft or some silent trade war going on between China and the US. It's certainly not because the US gov is "shocked" that China uses its supercomputers to do nuclear research.
Then comes Machiavellian management strategies that demonstrate large companies are not idiots, and withdrawal of such scheme 2-5 years later, while still holding sales lines in China.
Source: Have worked on both sides of the technology transfer equation.
It might be hard for the U.S. government to effectively prevent the Chinese government from doing these simulations, but it's plausible to me that the U.S. government would still want to see the Chinese government hindered in nuclear weapons simulations.
No, seriously.
1. Intel loses this business.
2. US employees lose their job.
3. China acquires skills designing the system themselves.
4. And they get to have their supercomputer after all.
How can anyone not see how stupid these export restrictions are?
2. Augments slow CPU with impossible to programming Xeon Phi.
3. Spends significant effort achieving heroic linpack benchmark.
4. Researchers are unable to obtain real world fast compute.
5. Chinese programs for stealth, submarines, nuclear are significantly delayed. Expertise diverted into useless PR project.
6. PR backlash as US goaded to reinvest in next generation supercomputing, GPUs, etc.
Mission accomplished.
https://en.wikipedia.org/wiki/Tianhe-2
What is interesting about this new supercomputer is that it is apparently built with CPUs that were developed and produced by the Chinese themselves.
New Supercomputer: https://en.wikipedia.org/wiki/Sunway_TaihuLight
CPU architecture: https://en.wikipedia.org/wiki/ShenWei
So that Chinese number seems awfully low to me. I would expect that the number Amazon has to be higher, for instance. Same for Microsoft and Google. Therefore I'd be amazed if the DoD, NSA and even the DoE wouldn't have more capacity available.
When I ran Exacycle, we distributed protein folding, protein design, drug discovery, telescope design, and other problems globally. We never had an issue with latency, because these problems all partition really well. People who claim supercomputers are "necessary" for these problems typically construct problems that are well-matched to supercomputers (for example, running molecular dynamics on huge proteins) but they tend not to have very high scientific value.
In my experience, partitioning to minimize communication has always increased my total scientific throughput, while programming to supercomputers has always reduced it.
Compared to Tianhe-2, the previous top system, 2.7x more flops, and 3.2x more flops per watt.
A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2. That kind of makes a supercomputer look like a pile of hardware, no matter how well it's organized or engraved.
[1] http://www.hpcwire.com/2016/06/19/china-125-petaflops-sunway...
Note that K computer (#5 system, Japan) scores 4.9% for HPCG. So Tianhe-2 and Titan are again 4x worse for HPCG compared to systems which score best for HPCG.
I don't think it's as simple as LINPACK bad, HPCG good. LINPACK is representative for some workloads, when compute dominates. HPCG aims to balance compute and memory. There is Graph500, if memory dominates for your workload.
By the way, K computer is #1 in Graph500.
The interconnect on the Sunway TaihuLight seems to be standard Infiniband, from reading other articles about the system.
Kind of reminds me of when we heard a rumor than a rival ISP had bought a handful of Sun servers (in the year 2000). We thought they had tends of thousands of customers, figured they would be launching DSL, figured that had thousands of web hosting customers...
Turns out they got some weird grant so they spent the cash on those, otherwise they would have lost the money. They went out of business a year later and we kept chugging along with our Pentium III servers.
There are enough high quality universites in China to make use of it. I can not imagine this is true. There may have not been one big algorithm that it was built for, but university researchers always have stuff to compute.
[1] https://learncryptography.com/cryptocurrency/51-attack
Yes, until one starts to lose on most such trivial metrics, and then they have a more substantial problem than losing on trivial metrics...
Those things can be long term signs about where the future is going.
>A while back I recall reading rumors that China didn't even know what to process on their Tianhe-2.
Yeah, those yellow people are so ignorant -- it takes white folks to show them what to process in their supercomputers. Sorry, but those rumours sounds openly racist.
I'd be interested to hear from someone who knows/has experience, whether any of the top 100 super computers ever have idle windows.
Having done exclusively HPC for almost 16 years now, I can state that there is one and only one immutable law of HPC: grad students will find a way to squeeze out every second of available CPU time on a large system. The amount of effort going into taking advantage of those empty spots in the clusters where smaller jobs can run quickly while the scheduler drains resources for a large job (called backfilling) is tremendous.
The thing that sticks with me is what a fantastically complicated problem HPC job scheduling is. I've seen dozens of fresh-faced undergrads or first year grad students come in with a full head of steam and decide that they're going to "solve" HPC job scheduling, but every single one has gotten bogged down in the minutiae and the muck, and I've never seen software that "gets it right". Every scheduler sucks in its own unique way.
Whether all of those jobs need an old-fashioned supercomputer is certainly up for debate, but they certainly get used, at least in my experience.
Or was it a combination of that with "well its there, and we want to do this thing that would happen quicker on the big machine" - IE not necessarily enabling new research, just adding a nice speedup to existing work?
So no, these machines are often used for problems which really do utilize the large-scale nature of the computers.
So lots of the "big data" type problems with lots of uncoordinated jobs aren't going to take advantage of the primary component of these big computers, the node interconnect. The computers really shine with big fluid dynamics simulations where there's lots of chatter between different components of the simulation to communicate boundary conditions and the like.
>> they certainly get used, at least in my experience
As a taxpayer, this made me feel good, thank you for sharing.
That applies to most of supercomputers everywhere. "If you build it, they will come".
Besides the knowledge attained in building supercomputers ( which all major nations should know how to do ) and processor technology, the added benefits and new uses of these machines will come later.
The internet was created before people really knew what to do with it. The same thing with computers. The same thing with everything else.
What matters most might be that China is maturing CPU technology that isn't nearly as congenial for US TLAs to hack in to. It might not matter so much for this supercomputer, but if Chinese suppliers put price ahead of things like opaque management processors, it might lead to tipping the balance away from easily exploitable endpoints.
The following paper compares the EU and US ensembles: http://journals.ametsoc.org/doi/full/10.1175/MWR2905.1
Is there plans to commercialize these chips? We need accessible low cost ultra-high core count chips widely available. I feel that Intel and AMD have been under performing in this area (growing core count) for the last decade. And Intel has kept the price of CPUs exceptionally high for the last decade as well.
If low cost ultra-high performance chips from China under cut Intel's excessively priced Xeons (+$3K per CPU), it really could change things. If these start to spread, I wouldn't be surprised if Intel tried to prevent the spread of these CPUs outside of China via trade barriers.
Every dollar spent on building these computers is a dollar NOT spent in the budget of a research-driven grant. Give these dollars to physicists, mathematicians, genomics scientists, chemists, climate people etc. This way the demand in computing will dictate the type of systems built.
p.s. I've attended the event in person that's listed in the article hosted by the OSTP, which is a part of the White House (National Strategic Computing Initiative). My conclusion was most of these computers aren't needed and a misdirection of funds to the tune of billions of dollars that computing / scientific computing could really benefit from in other ways.
If it's not an MPI job, it's going to be painful.
What are you using for communication between nodes?
To make the tone clear I'm genuinely interested as I'm not sure what running on a Cray prevents you from doing?
We don't need to communicate between the nodes; We don't need MPI. We need distributed databases and we might need Spark. We need machine learning. We need servers. We would like availability and reliability.
We get full batch nodes that are relatively anemic in memory for our workload (4GB/core) for a maximum of 24 hours at a time.
Which just emphasizes the point that most applications would be better served by cheaper, more resilient hardware.
http://www.eetimes.com/document.asp?doc_id=1261622
Huh. At the core of almost every problem I've ever encountered in my 20+ year career programming that I would consider "interesting" is the need to allocate work across many compute nodes that have some amount of shared state. You could just as well take A* as a baseline for these types of problems. Your typical cloud computing infra/design is not a good match for these.
It's true that we need shared data, but we don't need shared state (memory) for any of our workloads, and shared data (e.g. disk/db) is a much easier problem to solve than shared state.
There is still coordination and orchestration, but that's an extremely coarse amount of communication compared to the cost of Cray interconnects. That being said, there are cases where we might benefit from the Cray networking, but that comes at the cost of other tradeoffs (no local disk, low memory per core).
So what do we do? Well, we use a handful of Ferraris to get the job done because they are available when cheap bus would have suited us fine. The double whammy is that the Ferraris end up in the shop all the time and occasionally somebody else gets exclusivity to them when they want to get in a Bell prize submission.
Or you can send get the network admins to allow you to send them a email, but the first suggestion sounds more fun :P
1) I have a rail system that I want to optimize train movement through the topology. My search space is the number of running locomotives to the power of the number of switches in the system. On a continental rail system, that's a number with many, many, many zeros after it. To optimize this, I have many cores chewing through the space, but to know where to search they must share a visited node cache otherwise they will search each others space and waste effort.
2) I have a system that predicts part failures by collecting sensor data of the machine in a time series. I must look for patterns of change in these sensor readings and compare that to patterns that I know about, often drilling through data over many weeks looking for feature alignment in the sensor matrices. Same problem: many cores, some amount of space that is known vs. unknown that must be shared, but as a result is a contentious resource.
On the other hand, you can just run N independent simulations, and pool the statistics. In many cases, this is as good as if not better than running one long MD simulation. Here, all the processors share a common input file, but it's a single file, it only has to be communicated once. Any results from the processors are compacted, and sent to a large file server, which in itself is designed with partitioning since all the output files are independent. This approach won't let you work with large systems (unless your individual servers have more RAM), but it scales linearly to far more processors than tightly coupled parallelism, for far cheaper, and often, the scientific results are as good or better.
This shows up in many cases- in my 20+ year career programming, nearly all the problems I saw people run on supercomputers eventually could be run on CPUs with easier code and better performance. There are some irreducible codes for which only supercomputers are capable of meeting the required specs, and those are the only codes which should run on supercomputers. Otherwise you're pissing money on interconnect and hard to program environments).
In short: if you can convert your supercomputer job into a cloud cluster, you should do so, because it will save time and money. If you can't, then you should use a supercomputer.
Note also that cloud interconnects have gotten significantly better over the years- for me, 100Gbit throughput and latency (as long as the cluster has the total parallel bisection bandwidth) is fine for me and most other people.
/I work for Basho/
What's the evidence for this claim? AFAIK the DoE INCITE awards -- which is how you get allocated time on the big supercomputers -- are quite competitive.
Also the supercomputers don't cost billions. Aurora, for example only costs $200 million, and if it lasts as long as its predecessor Mira did, that is $200 million spread over 6 years, which is a pretty small number.
5 Preliminary progress of scientific computing applications on the TaihuLight
5.1 Refactoring the community atmospheric model (CAM) on the Sunway TaihuLight
5.2 A fully-implicit nonhydrostatic dynamic solver for cloud-resolving atmospheric simulation on eight million cores
5.3 A highly effective global surface wave numerical simulation with ultra-high resolution
5.4 Peta-scale atomistic simulation of silicon nanowires
5.5 Large-scale phase-field simulation for coarsening dynamics based on Cahn-Hilliard equation with degenerated mobility
5.6 Application: summary and comparison
Sierra and Summit are coming out soon, which are POWER based.
https://www.olcf.ornl.gov/summit/
http://www.cnet.com/news/ibm-nvidia-land-325-million-superco...