It's interesting that Dell won this one. Sadly there's not a lot of technical info about the system itself or the big storage expansion (probably more Lustre, sadly - my tax dollars at work).
Lustre nominally scales higher than GPFS, but GPFS is more reliable and better performing for anything having to do with metadata. It's also a lot more expensive license-wise. There's other competition, but it's not proven at these scale levels. Top 500 deals are bad business so it's not really worth it to develop/test for them thus leaving the business to the incumbents (e.g. the dumpster fire that's IBM or the Arkham Asylum that's Lustre).
Lustre at scale is a disaster, but one you can pay for people to help you with. GPFS is unbelievably expensive, but is a lot more reliable wrt metadata, but it is also quite fragile.
In my original question, I mean other than the obvious one of Lustre.
Yes, but EMC doesn't do that level of storage. For HPC, Dell resells various hodgepodges of their servers running Lustre (or rarely GPFS) with JBOD or cheesy RAID for the disks themselves.
I understand that, because those are basically the only filesystems anyone could use for this application. The comment was aimed at the "various hodgepodges of their servers ... with JBOD or cheesy RAID for the disks themselves" part.
But Isilon is purely NFS (and/or SMB/CIFS) and doesn't support the parallel access that GPFS or Lustre do. It's not really a choice for this workload. Plus it'd be a lot more expensive than HPC storage - enterprise level RAS (reliability, serviceability, availability) costs. I've seen comparable capacity Isilon & DDN quotes (w/GPFS or Lustre) and they're worlds apart.
Has there been a rash of failures? Not a meteorologist, but I usually hear that ECMWF & GFS have generally been trading blows, and have different strengths.
there are a few high profile cases where the Euro predicted hurricane paths better then the GFS. Granted these hurricanes are HUGE events and better hurricane forecasts saves lives etc. However these models predict ALL the weather, all the time, not just hurricanes. We use the GFS (in combination with other models as well) to shape our forecasts every day for all kinds of weather and millions across the country rely on these data.
The guy reading you a forecast on news channel ? If they are providing a forecast for greater than 3 days in the future, you bet they are including the GFS info in their predictions!
The GFS was bad at tropical weather this year. Though I have to say, the GFS has been a little less crazy in long term forecasts on the US West Coast lately.
Not sure how much processing power will improve the tropical storm forecasts, though. I had the impression that had to do more with the model's physics. Though it would be nice to catch up with the ECMWF's data assimilation prowess, that would definitely help.
It will be interesting to see what happens with weather modeling with the new Panasonic model. Although getting any info about it at all has been incredibly difficult since its all private.
The GFS though is the best public domain model in the world. ECMWF is proprietary, both code and model outputs, and the raw data are only accessible through paid agreements.
There is a crippled version of IFS (the ECMWF forecast model) which has the data assimilation code removed available [0], thought its licence terms are pretty restrictive for most uses [1] besides pure research (it is, for example, forbidden to use it for personal uses!).
I've been trying to figure out api access to forecasting data from ECMWF
Do you happen to know if wind speed forecasting data requires a commercial license? Seems like the free api key only gives you access to historical data?
32 comments
[ 0.30 ms ] story [ 56.2 ms ] threadLustre at scale is a disaster, but one you can pay for people to help you with. GPFS is unbelievably expensive, but is a lot more reliable wrt metadata, but it is also quite fragile.
In my original question, I mean other than the obvious one of Lustre.
The guy reading you a forecast on news channel ? If they are providing a forecast for greater than 3 days in the future, you bet they are including the GFS info in their predictions!
Not sure how much processing power will improve the tropical storm forecasts, though. I had the impression that had to do more with the model's physics. Though it would be nice to catch up with the ECMWF's data assimilation prowess, that would definitely help.
It says that Panasonic posted the data from this online so scholars could evaluate it, and gives a Google drive link, but that's 404 now.
[0] https://software.ecmwf.int/wiki/display/OIFS/ [1] https://software.ecmwf.int/wiki/display/OIFS/OpenIFS+Licensi...
Do you happen to know if wind speed forecasting data requires a commercial license? Seems like the free api key only gives you access to historical data?
Looks like they shut it down April 2016.
https://rdhpcs.noaa.gov/
https://www.ncrc.gov/