In a (simplified) nutshell: We can fit more transistors in a given unit of area because we make them smaller. That used to just mean we increased clock frequencies (make it faster) but comparatively recently (decade or two) meant we increased parallelism.
Moore's Law is expected to fail because we are now reaching the point where smaller transistors are very heavily impacted by actual limitations imposed by physics.
So while it is possible we'll have a technology shift and see similar performance gains, it won't really be Moore's Law anymore (unless we start using Pym Particles or something).
There is some work regarding making transistors out of different materials as a way to eke a bit more out. Similarly, there is a lot of work regarding layering circuitry to an even greater degree. And, of course, there is the usual pie in the sky solutions.
But none seem all that promising and my gut is that we'll focus more on interconnects and algorithmic improvements.
Computing using coupled magnetic spin. Or photonics. Or nanomechanical rod logic. Or nano-electro-mechanical logic. Or ballistic electrons. Or single electrons in nano-structures similar to what a cell uses in the Krebbs cycle.
The economic implications of Moore's Law are quite clear to me. Roughly one observes that every 18 months the expected computing power one can buy per unit currency doubles. Of course this is predicated upon physical possibility. 32 years would imply another 21 process halving or so, and that would take features down from 14 nm to an untenable 0.7 nm. However, if you have a better estimate, I'd be glad to hear it.
Yes. Economically, my bank account gets larger every year. Ergo, I must be saving up all my birthday cash
I don't have an estimate. My point was simply to explain to you why your logic was flawed as we are nearing the limits of what Moore's Law can give us without some pretty massive changes. This isn't a case of "Clock speeds are capped. We are doomed. Oh, wait, we can just put two slower ones on the same die" and is more "So... we are out of physical space..."
Trends are great when you are trying to make sense of data and estimate how to move forward. But they should not be used in a manner that ignores actual data.
Note that 21 halvings doesn't take you from 14nm to 0.7nm, but rather to about 7 femtometers, or about 0.000007nm. For comparison, that's roughly 10x the size of a single proton.
I'd say a better estimate would be to assume density stops increasing around the point when feature size is the size of a silicon atom. I'm sure that'll be way off, but closer than estimating 21 more doublings.
It also ignores that we're likely entering a period of anti-rational and anti-science thought, technology may have a few pull-backs before continuing its rise.
I think this is a relatively mild period of anti-science. Religion is largely on the decline. Fake news is being called out as "fake". College educations are valued even by those without them in general, and the job market certainly values them.
I think it seem like there is a much larger anti-science sentiment then there is because these people have been given a fresh voice with social media and for the first time in a long time they can connect with each other and build echo chambers to shout at eachother in.
Some of this spilled out in the last election and provided a non-trivial number of votes for a candidate who was clearly a demogogue, instead of voting for a different demogogue who appealed less to the uneducated.
Mainframes are great, already using almost[0] memory safe systems programming language on the 60's with Burroughs, followed by IBM and a few other vendors.
Virtualization and containers with the 360.
Bytecode as universal binary format with JIT/AOT at kernel level, DB based file system, System/38 and AS/400.
Object based OS, AS/400.
[0] - They still have the issue of leaks and double free though, but everything else is safe Algol style with explicit unsafe blocks/modules required.
Talk about a commitment to backwards compatibility – you can run binaries built 30 years ago for god knows what proprietary processor on a modern POWER8 system without recompiling.
I also find it kind of amusing that IBM, a primarily consulting company, developed AS/400, given that part of its sales pitch is that integrated database requires no maintenance and you can forget entirely about your IBM i and just leave it running for a decade.
It's a neat system. I wish I had the opportunity to use one. In many ways it feels like we're still catching up to what System/38 was doing in 1979.
I was responsible for doing backups on a AS/400 during a Summer internship in the early 90's. Sadly did not do much more than exchanging the tapes, logging in and starting the backup.
> "Wish there was a good book about the history of crays. It's the kind of thing you hear talked about but are unlikely to ever see."
That would be a great read.
My favorite Cray tidbit: for fun, Seymour Cray dug tunnels underneath his home, and had a lot of his breakthroughs while doing so.
he attributed the secret of his success to "visits by elves" while he worked in the tunnel:
"While I'm digging in the tunnel, the elves will often come to me with solutions to my problem."
I think it's speeding up. 1975 Cray-1 had approximately same 160 MFlops as first iphone had. And it was 32 years between them. So, probably we'll have 130 petaflops around 2033-2035? :)
For many services the phone is just a relatively dumb front end, and the actual computing happens on cloudy clusters elsewhere.
Given that Moore's Law is creaking I wouldn't expect pocket petaflops any time soon. I'd expect a serious outbreak of cloudy clusters everywhere, and perhaps a dynamically reconfigurable Internet 2.0 with completely transparent non-localised computation.
This might change if computing finally goes optical and/or quantum. But if we're pushing electrons around wires, current hardware is close to the physical limits. The only way to speed it up is to build a lot more of it and speed up the connections.
Note that these measures are biased by bandwidth. CPUs today have massive peak FLOPS, but limited sustained bandwidth, while the Cray being designed with a very large one to ensure regular performance.
You are probably overestimating the Cray, modern CPU's have a lot of bandwidth. And due to being physically smaller fewer issues with latency, and fairly large on chip cache.
Someone did the math and a Cray was on par with an intel core i3 series (early gen) in sustained perf even though in terms of raw GFLOPS the intel cpu was two leagues above. That's all I'm saying.
That's not a Crey 2. That is talking about the X-MP it's successor.
https://en.m.wikipedia.org/wiki/Cray_X-MP "The Cray-2, a completely new design, was introduced 1985. A very different compact four-processor design with from 64 MW (megaword) to 512 MW (512 MB to 4 GB) of main memory, it was specified to 500 MFLOPS but was slower than the X-MP on certain calculations due to its high memory latency.
The X-MP-succeeding Cray Y-MP series was announced in 1988; it also had a new design, replacing the 16-gate ECL gate arrays with a more compact VLSI gate array with larger circuit boards. It was a major improvement of the X-MP supporting up to eight processors."
Note latency is huge for these systems meaning for most workloads modern cellphones absolutely crush them.
It's just too much computing power – there will come a time when many things have enough.
If you look at Apple's recent offerings, it would seem they think that time is more or less here.
I certainly almost never use the full computing power of anything I'm using – the limiting factors aren't hardware any more but the software running on it.
We're still coming up with everyday tasks/usage that require more and more power, storage and internet bandwidth. First everything was text. Then we started using photo/audio, then we started using low res video, then video resolution kept enlarging (though recently this slowed down, we got to 1080p pretty fast, switching to 4k isn't as fast), now we invent VR, probably some 3d/holograms format will follow. I don't know what can come after that, maybe we'll finally will have enough power/bandwidth/storage :)
We're reaching the flat part at the top of the sigmoid innovation curve. We're still using 1960s technology. It's just been through a good few generations of refinement.
VR might just about squeeze through now, but given the hardware limitations - and the fact that people look really dorky using it - I'm not expecting it to drive a new explosion of user interest.
We really need some completely new tech to drive a new wave of innovation. The obvious candidates are optical/quantum and perhaps direct neural interfacing. Both are still science fiction, but that may change by 2030.
More extreme technologies may also be possible, but they're beyond speculative.
For now it may be useful to remember that technology rarely develops linearly, so speculating about future CPUs is like speculating about the future of transatlantic cruise liners, while ignoring the fact that someone somewhere is working on heavier than air flight.
you might not, but many gamers, streamers, audio/video creators create content today, often 4K (or even higher) which requires a lot of processing power and internet bandwidth, so assuming that demand will not go up is a bit naive. For everyday tasks, sure what we have is fine, but as soon as you move into producing or consuming complex media (which is more and more mainstream these days) i don't think it will stop anytime soon.
I guess that's theoretically possible, according to Landauer's principle [1]. It's hard to say if it will ever be feasible. It feels like we are running into some serious limitations, since CPUs don't seem to be getting any faster. The CPU will probably have to be totally redesigned from the ground up, using different chemicals, and perhaps light instead of electricity. Or maybe some insane molecular structure computed and created via genetic algorithms and machine learning. Evolution did it to form our brains, so it's definitely possible.
I'm not sure how to do the calculation to find the minimum amount of power required for 130 petaflops, but it seems like it's well within the capacity of today's mobile batteries.
We don't need that much power to run our Facebook, Camera and Mail apps. Really, we have fast enough hardware, good enough screens and headphones. You can't hear over 16khz and can't see over 400dpi, so they can't improve image and sound any more.
What we need to advance is specialized Deep Learning cores that are compact, fast and low power, so they don't kill the battery. On the other hand, for general apps it's much less necessary to make the CPU faster.
Because it's a government project. We give you a price and specs, you build it late with cost overruns, then we realize the specs weren't really what we wanted.
Sunway TaihuLight (the most powerful supercomputer right now) was benchmarked to have 93 petaflops[1].
The cost was 1.8 billion Yuan (US$273 million). So cheap I guess.
Serious if likely naive question: what's the use case for supercomputers these days, instead of just spinning up a cluster in your favorite cloud and running your favorite MapReduce derivative on it? These guys pulled off a petaflop on AWS with 156k cores for $33k, and that was in 2013.
It's actually the opposite - heavily parallelizable calculations can easily be split across physically separated machines. Supercomputers are necessary for computations that are hard to parallelize, ie those where there's a lot of data dependency between individual subproblems and data transfer between computing nodes is a bottleneck.
There's plenty of reasons. Here's a bullet list with some thoughts.
* Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day.
* Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR.
* Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality.
* MR isn't an ideal paradigm for every calculation. MR comes from
search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you.
* Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat.
* A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine.
Also, the page says it's only a theoretical petaflop:
"While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."
The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry.
There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
It depends on the problem you want to solve. Working hard on detail oriented work and not giving up until you're successful is always a transferrable experience. ut a lot of the data analysis systems at the moment seem to want an SQL interface. It's a good idea because it's a stable language that lots of people already know. But your work with MPI won't really help you directly.
>There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it.
Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.
This is a fantastic answer. Could you or anyone else please point me to any examples of trying to run high-resolution weather forecasting, such as COAMPS [0] or WRF [1] on commodity cloud infrastructure instead of HPC?
Googling turns up some hits, but not much. The basic issue is that weather prediction is one of those problems that can always use more computational power. It is even a known problem that the models used by the National Weather Service are inferior to the models used by agencies in Europe, and one of the things holding us back is raw computational power. From "Why Isn't the U.S. Better at Prediction Extreme Weather?" (http://www.nytimes.com/2016/10/23/magazine/why-isnt-the-us-b...):
While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out.
Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load.
HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.
I think the only thing that most HPC sites use which wouldn't be found in cloud infrastructure is the interconnects and a parallel file system. Back in 2004 I guess HPC sites were often using PPC on AIX. That's just gone. It's almost all Linux on x86_64 now aside from some interesting machines in Japan and China.
So the distinction of "commodity" that Google made in the original map reduce paper in 2004 is gone. And the benefits of data locality are all but gone. The paper was great when it came out but it's long in the tooth since the hardware of today is different.
To answer your actual question, I've never heard of anyone running WRF on cloud infrastructure. It doesn't have to be a terrible idea. If you're trying to come up with a competing model then the agility you get might be needed. i.e. being able to roll onto new hardware as it becomes available instead of having to work with budget cycles for a hardware refresh lets you move from tesla k80 to titans whenever AWS decided to make them available.
Searching around, I see Glenn Lockwood did some benchmarks in 2013 with AWS:
I think the prevailing issue is that latency on virtualized nodes is poor so you have to work around it. Maybe fat nodes with gpgpu can overcome the issues.
If you need to talk to me privately then you can message me on reddit.
I know that New Zealand's MetService found that WRF running AWS is competitive, but I don't know if they published the results. Can put you in touch with their main point of contact for this kind of work.
Different kinds of applications. Think of it like public transportation versus owning a car.
For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle.
Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving.
And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations.
Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
Note that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale above 128 nodes on typical supercomputers due to the communication requirements. I switched to running N independent simulations and pooled the stats, getting far more high quality results.
I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches.
Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
That's a good question. There are a small number of people who have experience with it. The number is small because supercomputers don't really provide any speedup advantages, although this is mainly because the code developers of distributed DNNs have been using TCP/IP networks with standard RPC instead of fast interconnect.
I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI.
There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.
Great question. I have a friend who used to work on the human brain project and I asked him a similar question. What surprised me is that the software that gets run on super computers is usually quite simple BUT the computational complexity is high or the volume of data is large. Fairly non-technical people (i.e. not necessarily computer scientists but maybe mathematicians) would typically submit small programs written in Python that would be automatically compiled into something that used the hardware to its maximum potential (instead of compiling it to something more portable). I believe that software running on super computers is closer to the wire than clustered computers. The operating systems are often very stripped down and custom too.
pulled off a petaflop on AWS with 156k cores for $33k
They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day.
In just 30 days on AWS, you could buy the $173m supercomputer in the article.
Others have mentioned practical issues like performance but there's also cost. When you're actually using a lot of capacity and doing so pretty consistently, Amazon is extremely expensive compared to dedicated hardware.
HPC at centers like NCSA have a much lower latency interconnect than what is used by Amazon, Google, etc. and there are some computational problems that require the lower latency interconnect.
Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.
For one, getting data into AWS is a problem for scientific computing. For example, the Large Hadron Collider experiment generates 27 terabytes of data per day and has its own network: https://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid
Every time I see one of these headlines, I think "so what?". Are we just awed by how many "flops" it has or is there some actual practical advancement we should be looking forward to?
Does it worry anybody that China owns the top two positions and the US supercomputers in places 3-6 aren't even close to the top two in terms of performance?
87 comments
[ 3.8 ms ] story [ 162 ms ] threador - how long ago would $173M have bought you the kind of power we carry around today in a pocket?
In a (simplified) nutshell: We can fit more transistors in a given unit of area because we make them smaller. That used to just mean we increased clock frequencies (make it faster) but comparatively recently (decade or two) meant we increased parallelism.
Moore's Law is expected to fail because we are now reaching the point where smaller transistors are very heavily impacted by actual limitations imposed by physics.
So while it is possible we'll have a technology shift and see similar performance gains, it won't really be Moore's Law anymore (unless we start using Pym Particles or something).
What kind of technology shift do you mean? Like a totally different computing paradigm?
But none seem all that promising and my gut is that we'll focus more on interconnects and algorithmic improvements.
But time will tell.
For those reading along, this is a fictional particle named after Hank Pym - AntMan - from Marvel comics. It's not a technology in a lab somewhere.
I don't have an estimate. My point was simply to explain to you why your logic was flawed as we are nearing the limits of what Moore's Law can give us without some pretty massive changes. This isn't a case of "Clock speeds are capped. We are doomed. Oh, wait, we can just put two slower ones on the same die" and is more "So... we are out of physical space..."
Trends are great when you are trying to make sense of data and estimate how to move forward. But they should not be used in a manner that ignores actual data.
I'd say a better estimate would be to assume density stops increasing around the point when feature size is the size of a silicon atom. I'm sure that'll be way off, but closer than estimating 21 more doublings.
I think it seem like there is a much larger anti-science sentiment then there is because these people have been given a fresh voice with social media and for the first time in a long time they can connect with each other and build echo chambers to shout at eachother in.
Some of this spilled out in the last election and provided a non-trivial number of votes for a candidate who was clearly a demogogue, instead of voting for a different demogogue who appealed less to the uneducated.
Estimates put the processing power of an iPhone 5 at at around 2.7x times that of the Cray 2[2]
[1] http://www.theregister.co.uk/2012/03/08/supercomputing_vs_ho...
[2] http://pages.experts-exchange.com/processing-power-compared/
Mainframes in general too.
I'm currently listening to Soul of a New Machine.
Mainframes are great, already using almost[0] memory safe systems programming language on the 60's with Burroughs, followed by IBM and a few other vendors.
Virtualization and containers with the 360.
Bytecode as universal binary format with JIT/AOT at kernel level, DB based file system, System/38 and AS/400.
Object based OS, AS/400.
[0] - They still have the issue of leaks and double free though, but everything else is safe Algol style with explicit unsafe blocks/modules required.
I also find it kind of amusing that IBM, a primarily consulting company, developed AS/400, given that part of its sales pitch is that integrated database requires no maintenance and you can forget entirely about your IBM i and just leave it running for a decade.
It's a neat system. I wish I had the opportunity to use one. In many ways it feels like we're still catching up to what System/38 was doing in 1979.
That would be a great read.
My favorite Cray tidbit: for fun, Seymour Cray dug tunnels underneath his home, and had a lot of his breakthroughs while doing so.
https://en.wikipedia.org/wiki/Seymour_Cray#Personal_life"Seymour Cray was a man of few words. I was there for three weeks before I realized he was not the janitor."
https://news.ycombinator.com/item?id=11941941
So, if this was linear, we can all expect 130 petaflop computers around 2043 for around $3,500?
Lots of caveats here though, things aren't usually as linear as all this, and this is very much a back-of-a-napkin calculation.
Given that Moore's Law is creaking I wouldn't expect pocket petaflops any time soon. I'd expect a serious outbreak of cloudy clusters everywhere, and perhaps a dynamically reconfigurable Internet 2.0 with completely transparent non-localised computation.
This might change if computing finally goes optical and/or quantum. But if we're pushing electrons around wires, current hardware is close to the physical limits. The only way to speed it up is to build a lot more of it and speed up the connections.
I'll link the url when I've scanned my bookmarks.
psedit: look at jojomonkeyboy's comment http://www.techrepublic.com/blog/classics-rock/the-80s-super... he says an i7 2600 (not an i3) has less sustained compute power.
https://en.m.wikipedia.org/wiki/Cray_X-MP "The Cray-2, a completely new design, was introduced 1985. A very different compact four-processor design with from 64 MW (megaword) to 512 MW (512 MB to 4 GB) of main memory, it was specified to 500 MFLOPS but was slower than the X-MP on certain calculations due to its high memory latency.
The X-MP-succeeding Cray Y-MP series was announced in 1988; it also had a new design, replacing the 16-gate ECL gate arrays with a more compact VLSI gate array with larger circuit boards. It was a major improvement of the X-MP supporting up to eight processors."
Note latency is huge for these systems meaning for most workloads modern cellphones absolutely crush them.
If you look at Apple's recent offerings, it would seem they think that time is more or less here.
I certainly almost never use the full computing power of anything I'm using – the limiting factors aren't hardware any more but the software running on it.
VR might just about squeeze through now, but given the hardware limitations - and the fact that people look really dorky using it - I'm not expecting it to drive a new explosion of user interest.
We really need some completely new tech to drive a new wave of innovation. The obvious candidates are optical/quantum and perhaps direct neural interfacing. Both are still science fiction, but that may change by 2030.
More extreme technologies may also be possible, but they're beyond speculative.
For now it may be useful to remember that technology rarely develops linearly, so speculating about future CPUs is like speculating about the future of transatlantic cruise liners, while ignoring the fact that someone somewhere is working on heavier than air flight.
I would really like to be excited by the prospect of neural interfacing, but all I can imagine is people catching computer viruses.
I'm not sure how to do the calculation to find the minimum amount of power required for 130 petaflops, but it seems like it's well within the capacity of today's mobile batteries.
[1] https://en.wikipedia.org/wiki/Landauer's_principle
What we need to advance is specialized Deep Learning cores that are compact, fast and low power, so they don't kill the battery. On the other hand, for general apps it's much less necessary to make the CPU faster.
[1] https://www.top500.org/news/china-tops-supercomputer-ranking...
[1] http://arstechnica.com/information-technology/2013/11/18-hou...
* Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day.
* Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR.
* Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality.
* MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you.
* Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat.
* A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine.
Also, the page says it's only a theoretical petaflop:
"While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."
There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
>There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it.
Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.
[0] http://www.nrlmry.navy.mil/coamps-web/web/home [1] http://www.wrf-model.org/index.php
And fnord123, could you please email me?
While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out.
Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load.
HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.
So the distinction of "commodity" that Google made in the original map reduce paper in 2004 is gone. And the benefits of data locality are all but gone. The paper was great when it came out but it's long in the tooth since the hardware of today is different.
To answer your actual question, I've never heard of anyone running WRF on cloud infrastructure. It doesn't have to be a terrible idea. If you're trying to come up with a competing model then the agility you get might be needed. i.e. being able to roll onto new hardware as it becomes available instead of having to work with budget cycles for a hardware refresh lets you move from tesla k80 to titans whenever AWS decided to make them available.
Searching around, I see Glenn Lockwood did some benchmarks in 2013 with AWS:
https://glennklockwood.blogspot.be/2013/12/high-performance-...
This lot seem to have done some more benchmarking in 2014 (tldr 44% overhead on VMs - but take the time to read it since glockwood is always a good read): https://www.researchgate.net/publication/301407792_Performan...
I think the prevailing issue is that latency on virtualized nodes is poor so you have to work around it. Maybe fat nodes with gpgpu can overcome the issues.
If you need to talk to me privately then you can message me on reddit.
Abstract is here: http://www.eresearch.org.nz/event/eresearch-nz-2013/do-hpcs-...
For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle.
Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving.
And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations.
Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches.
Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI.
There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.
They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day.
In just 30 days on AWS, you could buy the $173m supercomputer in the article.
This isn't a serious alternative.
Anime rendering? pretty rad action scenes!
Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.
all algorithms will become O(1) and even the shittiest code will be instantaneous. which is great for me.
great for me maybe bad for encryption and governments and idk.
Right now, I guess, 1 GFLOP is about 0.5-1W of power. This means that 130PFLOPS computer will draw 75-130MW, definitely even more.
It can be a problem.