I continue to be a little confused by the distinction between Google, Google Research and DeepMind. Google Research, had made this announcement about 24-hour forecasting just 2 weeks ago:
https://blog.research.google/2023/11/metnet-3-state-of-art-n... (which is also mentioned in the GraphCast announcement from today)
DeepMind recently merged with the Brain team from Google Research to form `Google DeepMind`. It seems this was done to have Google DeepMind focused primarily (only?) on AI research, leaving Google Research to work on other things in more than 20 research areas. Still, some AI research involves both orgs, including MetNet in weather forecasting.
In any case, GraphCast is a 10-day global model, whereas MetNet is a 24-hour regional model, among other differences.
Good explanation. Now that both the 24-hour regional and 10-day global models have been announced in technical/research detail, I supposed there might still be a general blog post about how improved forecasting is when you search for "weather" or check the forecast on Android.
IIRC the MetNet announcement a few weeks ago said that their model is now used when you literally Google your local weather. I don't think it's available yet to any API that third party weather apps pull from, so you'll have to keep searching "weather in Seattle" to see it.
It's also used, at least for the high resolution precipitation forecast, in the default Android weather app (which is really part of the "Google" app situation).
Most likely explanation would be that Weather.com signed a contract with Google X years ago to have something placed there, and nobody wants to do the work to do anything about it.
MetNet-3 is not open-source, and the announcement said it's already integrated into Google products/services needing weather info. So, I'd doubt there's anything like a colab example.
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
That is not strictly true. The weather at time t0 may affect non-weather phenomena at time t1 (e.g. traffic), which in turn may affect weather at time t2.
Furthermore, a predictive model is not working with a complete picture of the weather, but rather some limited-resolution measurements. So, even ignoring non-weather, there may be local weather phenomena detected at time t0, escaping detection at time t1, but still affecting weather at time t2.
Interesting indeed, only one lagged feature for time series forecasting? I’d imagine that including more lagged inputs would increase performance.
Rolling the forecasts forward to get n-step-ahead forecasts is a common approach. I’d be interested in how they mitigated the problem of the errors accumulating/compounding.
I don't know much about weather prediction, but if a model can improve the state of the art only with that data as input, my conclusion is that previous models were crap... or am I missing something?
It's worth pointing out that "state of the weather" is a little bit hand-wavy. The GraphCast model requires a fully-assimilated 3D atmospheric state - which means you still need to run a full-complexity numerical weather prediction system with a massive amount of inputs to actually get to the starting line for using this forecast tool.
Initializing directly from, say, geostationary and LEO satellite data with complementary surface station observations - skipping the assimilation step entirely - is clearly where this revolution is headed, but it's very important to explicitly note that we're not there yet (even in a research capacity).
Yeah current models aren’t quite ready to ingest real time noisy data like the actual weather… I hear they go off the rails if preprocessing is skipped (outliers, etc)
I've been following these global ML weather models. The fact they make good forecasts at all was very impressive. What is blowing my mind is how fast they run. It takes hours on giant super computers for numerical weather prediction models to forecast the entire globe. These ML models are taking minutes or seconds. This is potentially huge for operational forecasting.
Weather forecasting has been moving focus towards ensembles to account for uncertainty in forecasts. I see a future of large ensembles of ML models being ran hourly incorporating the latest measurements
Not to take away from the excitement but ML weather prediction builds upon the years of data produced by numerical models on supercomputers. It cannot do anything without that computation and its forecasts are dependent on the quality of that computation. Ensemble models are already used to quantify uncertainty (it’s referenced in their paper).
But it is exciting that they are able to recognize patterns in multi year and produce medium term forecasts.
Some comments here suggest this replaces supercomputers models. This would a wrong conclusion.It does not (the paper explicitly states this). It uses their output as input data.
We have dozens of complementary and contradictory sources of weather information. Different types of satellites measuring EM radiation in different bands, weather stations, terrestrial weather radars, buoys, weather balloons... it's a massive hodge-podge of different systems measuring different things in an uncoordinated fashion.
Today, it's not really practical to assemble that data and directly feed it into an AI system. So the state-of-the-art in AI weather forecasting involves using an intermediate representation - "reanalysis" datasets which apply a sophisticated physics based weather model to assimilate all of these data sets into a single, self-consistent 3D and time-varying record of the state of the atmosphere. This data is the unsung hero of the weather revolution - just as the WMO's coordinated synoptic time observations for weather balloons catalyzed effective early numerical weather prediction in the 50's and 60's, accessible re-analysis data - and the computational tools and platforms to actually work with these peta-scale datasets - has catalyzed the advent of "pure AI" weather forecasting systems.
Great comment, thank you for sharing your insights. I don't think many people truly understand just how massive these weather models are and the sheer volume of data assimilation work that's been done for decades to get us to this point today.
I always have a lot of ideas about using AI to solve very small scale weather forecasting issues, but there's just so much to it. It's always a learning experience for sure.
It uses era5 data which is reanalysis. These models will always need the numerical training data. What's impressive is how well the emulate the physics in those models so cheaply. But since the climate changes there will eventually be different weather in different places.
Absolutely - but large ensembles are just the tip of the iceberg. Why bother producing an ensemble when you could just output the posterior distribution of many forecast predictands on a dense grid? One could generate the entire ensemble-derived probabilities from a single forward model run.
Another very cool application could incorporate generative modeling. Inject a bit of uncertainty in a some observations and study how the manifold of forecast outputs changes... ultimately, you could tackle things like studying the sensitivity of forecast uncertainty for, say, a tropical cyclone or nor'easter relative to targeted observations. Imagine a tool where you could optimize where a Global Hawk should drop rawindsondes over the Pacific Ocean to maximally decrease forecast uncertainty for a big winter storm impacting New England...
We may not be able to engineer the weather anytime soon, but in the next few years we may have a new type of crystal ball for anticipating its nuances with far more fidelity than ever before.
This is basically equivalent to NVIDIA's DLSS machine learning running on Tensor Cores to "up-res" or "frame-interpolate" the extremely computationally intensive job the traditional GPU rasterizer does to simulate a world.
You could numerically render a 4k scene at 120FPS at extreme cost, or you could render a 2k scene at 60FPS, then feed that to DLSS to get a close-enough approximation of the former at enormous energy and hardware savings.
I live in an area which regularly has a climate differently then forecasted: often less rain and more sunny. Would be great if I can connect my local weather station (and/or its history) to some model and have more accurate forecasts.
Because weather data is interpolated between multiple stations, you wouldn't even need the local station position, your own position would be more accurate as it'd take a lot more parameters into account.
One piece of context to note here is that models like ECMWF are used by forecasters as a tool to make predictions - they aren't taken as gospel, just another input.
The global models tend to consistently miss in places that have local weather "quirks" - which is why local forecasters tend to do better than, say, accuweather, where it just posts what the models say.
Local forecasters might have learned over time that, in early Autumn, the models tend to overpredict rain, and so when they give their forecasts, they'll tweak the predictions based on the model tendencies.
Interesting. So what I am looking for is probably an even more scaled down version? Or something that runs in the cloud with an api to upload my local measurements.
Hate to break it but one weather station wont improve a forecast? What are they supposed to do? Ignore the output of our state of the art forecast models and add an if statement for your specific weather station??
weather prediction seems to me like a terrific use of machine learning aka statistics. The challenge I suppose is in the data. To get perfect predictions you'd need to have a mapping of what conditions were like 6 hours, 12 hours, etc before, and what the various outcomes were, which butterflies flapped their wings and where (this last one is a joke about how hard this data would be). Hard but not impossible. Maybe impossible. I know very little about weather data though. Is there already such a format?
It's been a while since I was a grad student but I think the raw station/radiosonde data is interpolated into a grid format before it's put into the standard models.
This was also in the article. It splits the sphere surface in to 1M grids (not actually grids in the cartesian sense of a plane, these are radial units). Then there's 37 altitude layers.
So there's radial-coordinate voxels that represent a low resolution of the physical state of the entire atmosphere.
To call this impressive is an understatement. Using a single GPU, outperforms models that run on the world's largest super computers. Completely open sourced - not just model weights. And fairly simple training / input data.
> ... with the current version being the
largest we can practically fit under current engineering constraints, but which have potential to
scale much further in the future with greater compute resources and higher resolution data.
I can't wait to see how far other people take this.
It builds on top of supercomputer model output and does better at the specific task of medium term forecasts.
It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
I don't understand why this is downvoted. This is a classic thing to do with deep learning: take something that has a solution that is expensive to compute, and then train a deep learning model from that. And along the way, your model might yield improvements, too, and you can layer in additional features, interpolate at finer-grained resolution, etc. If nothing else, the forward pass in a deep learning model is almost certainly way faster than simulating the next step in a numerical simulation, but there is room for improvement as they show here. Doesn't invalidate the input data!
Because "iterative refinement" is sort of wrong. It's not a refinement and it's not iterative. It's an entirely different model to physical simulation which works entirely differently and the speed up is order of magnitude.
Building a statistical model to approximate a physical process isn't a new idea for sure.. there are literally dozens of them for weather.. the idea itself isn't really even iterative, it's the same idea... but it's all in the execution. If you built a model to predict stock prices tomorrow and it generated 1000% pa, it wouldn't be reasonable for me to call it iterative.
Could you point me to the part where it says it depends on supercomputer output?
I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs:
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
You can read about it more in their paper. Specifically page 36. Their dataset, ERA5, is created using a process called reanalysis. It combines historical weather observations with modern weather models to create a consistent record of past weather conditions.
I can't find the details, but if the supercomputer job only had to run once, or a few times, while this model can make accurate predictions repeatedly on unique situations, then it doesn't matter as much that a supercomputer was required. The goal is to use the supercomputer once, to create a high value simulated dataset, then repeatedly make predictions from the lower-cost models.
So best case scenario we can avoid some computation for inference, assuming that historical system dynamics are still valid. This model needs to be constantly monitored by full scale simulations and rectified over time.
TIL about Raspberry-NOAA and pywws in researching and summarizing for a comment on "Nrsc5: Receive NRSC-5 digital radio stations using an RTL-SDR dongle" (2023)
https://news.ycombinator.com/item?id=38158091
I don't using raw historical data would work for any data intensive model - afaik the data is patchy - there are spots where we don't have that many datapoints - e.g. middle of ocean... Also there are new satelites that are only available for the last x years and you want to be able to use these for the new models. So you need a re-analysis of what it would look like if you had that data 40 years ago...
In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data.
Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.
Extreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries.
Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.
Appreciate the response. Do you know of any services that provide what I described in the previous comments? I'm specifically interested in extreme weather conditions and their visual representation (hurricanes, tornados, hails etc.) with API capabilities
Go to:
nhc.noaa.gov/gis
There's a list of data and products with kmls and kmzs and geojsons and all sorts of stuff. I haven't actually used the API for retrieving these, but NOAA has a pretty solid track record with data dissemination.
KML files for storm tracks are still the best way to go. You could calculate storm tracks yourself for other weather models like DWD ICON, ECMWF IFS or MeteoFrance ARPEGE, but storm tracks based on GFS ensembles are easy to use with sufficient accuracy
Both APIs use weather models from NOAA GFS and HRRR, providing accurate forecasts in North America. HRRR updates every hour, capturing recent showers and storms in the upcoming hours. PirateWeather gained popularity last year as a replacement for the Dark Sky API when Dark Sky servers were shut down.
With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties.
Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.
They're very cautious about naming a "best" model though!
> Weather forecasting is a multi-faceted problem with a variety of use cases. No single metric fits all those use cases. Therefore,it is important to look at a number of different metrics and consider how the forecast will be applied.
I just quit photographing weddings (and other stuff) this year. It's a job where the forecast really impacts you, so you tend to pay attention.
The amount of brides I've had to calm down when rain was forecast for their day is pretty high. In my experience, in my region, precipitation forecasts more than 3 days out are worthless except for when it's supposed to rain for several days straight. Temperature/wind is better but it can still swing one way or the other significantly.
For other types of shoots I'd tell people that ideally we'd postpone on the day of, and only to start worrying about it the day before the shoot.
I'm in Minnesota, so our weather is quite a bit more dynamic than many regions, for what it's worth.
I would like to see an independent forecast comparison tool similar to Forecast Advisor, which evaluates numerical weather models. However, getting reliable ground truth data on a global scale can be a challenge.
Since Open-Meteo continuously downloads every weather model run, the resulting time series closely resembles assimilated gridded data. GraphCast relies on the same data to initialize each weather model run. By comparing past forecasts to future assimilated data, we can assess how much a weather model deviates from the "truth," eliminating the need for weather station data for comparison. This same principle is also applied to validate GraphCast.
Moreover, storing past weather model runs can enhance forecasts. For instance, if a weather model consistently predicts high temperatures for a specific large-scale weather pattern, a machine learning model (or a simple multilinear regression) can be trained to mitigate such biases. This improvement can be done for a single location with minimal computational effort.
It can take up to 10 min to generate a report - I had a spinner before but people just left the page. So I implemented a way to send it to them instead. I’ve never used the emails for anything else than that. Try it with a 10 min disposable email address if you like. Thanks for your feedback!
Ok, seems like your UI is not coming from a place of malice. However, pulling out an email input form at the final step is a very widespread UI dark pattern, so if nothing else please let people know that you will ask their email before they start interacting with your forms.
this is really cool, I've been looking for good snow-related weather APIs for my business. I tried looking on the site, but how does it work, being coordinates-based?
I'm used to working with different weather stations, e.g. seeing different snowfall prediction at the bottom of a mountain, halfway up, and at the top, where the coordinates are quite similar.
You'll need a local weather expert to assist, as terrain, geography and other hyper-local factors create forecasting unpredictability. For example, Jay Peak in VT has its own weather, the road in has no snow, but it's a raging snowstorm on the mountain.
This is great. I am very curious about the architectural decisions you've taken here. Is there a blog post / article about them? 80 yrs of historical data -- are you storing that somewhere in PG and the APIs are just fetching it? If so, what indices have you set up to make APIs fetch faster etc. I just fetched 1960 to 2022 in about 12 secs.
Traditional database systems struggle to handle gridded data efficiently. Using PG with time-based indices is memory and storage extensive. It works well for a limited number of locations, but global weather models at 9-12 km resolution have 4 to 6 million grid-cells.
I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube.
Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme.
While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours.
How did you handle missing data? I’ve used NOAA data a few times and I’m always surprised at how many days of historical data are missing. They have also stopped recording in certain locations and then start in new locations over time making it hard to get solid historical weather information.
I always suspect that they don't tell me the actual temperature. Maybe I am totally wrong but I suspect. I need to get my own physical thermometer not the digital one in my room and outside my house and have a camera focussed on it. So that later I can speed up the video and see how much the weather varied the previous night.
I see the limit for non-commercial use should be "less than 10.000 daily API calls". Technically 2 is less than 10.000, I know, but still I decided to drop you a comment. :)
I confirm, open-meteo is awesome and has a great API (and API playground!).
And is the only source I know to offer 2 weeks of hourly forecasts (I understand at that point they are more likely to just show a general trend, but it still looks spectacular).
Thank you, I didn't know!
I'd love to, but I'd need another 24 hours in a day to also process the data - I'm glad I can build on a work of others and use the friendly APIs :).
This is awesome. I was trying to do a weather project a while ago, but couldn't find an API to suit my needs for the life of me. It looks like yours still doesn't have exactly everything I'd want but it still has plenty. Mainly UV index is something I've been trying to find wide historical data for, but it seems like it just might not be out there. I do see you have solar radiation, so I wonder if I could calculate it using that data. But I believe UV index also takes into account things like local air pollution and ozone forecast as well.
I've never studied weather forecasting, but I can't say I'm surprised. All of these models, AFAICT, are based on the "state" of the weather, but "state" deserves massive scare quotes: it's a bunch of 2D fields (wind speed, pressure, etc) -- note the 2D. Actual weather dynamics happen in three dimensions, and three dimensional land features, buildings, etc as well as gnarly 2D surface phenomena (ocean surface temperature, ground surface temperature, etc) surely have strong effects.
On top of this, surely the actual observations that feed into the model are terrible -- they come from weather stations, sounding rockets, balloons, radar, etc, none of which seem likely to be especially accurate in all locations. Except that, where a weather station exists, the output of that station is the observation that people care about -- unless you're in an airplane, you don't personally care about the geopotential, but you do care about how windy it is, what the temperature and humidity are, and how much precipitation there is.
ISTM these dynamics ought to be better captured by learning them from actual observations than from trying to map physics both ways onto the rather limited datasets that are available. And a trained model could also learn about the idiosyncrasies of the observation and the extra bits of forcing (buildings, etc) that simply are not captured by the inputs.
(Heck, my personal in-my-head neural network can learn a mapping from NWS forecasts to NWS observations later in the same day that seems better than what the NWS itself produces. Surely someone could train a very simple model that takes NWS forecasts as inputs and produces its estimates of NWS observations during the forecast period as outputs, thus handling things like "the NWS consistently underestimates the daily high temperature at such-and-such location during a summer heat wave.")
I'm not sure why you're emphasizing that weather forecasting is just 2D fields. Even in the article they mention GraphCast predicts multiple data points at each global location across a variety of altitudes. All existing global computational forecast models work the same way. They're all 3d spherical coordinate systems.
See page three, table 1 of the paper. The model has 48 2D fields, on a grid, where the grid is a spherical thing wrapped around the surface of the Earth.
There is not what I would call a 3D spherical coordinate system. There’s no field f defined as f(theta, phi, r) — ther are 48 fields that are functions of theta and phi.
I read a decent amount of the paper, although not the specific details of the model they used. And when I say I "never studied" it, I mean that I never took a class or read a textbook. I do, in fact, know something about physics and fluids, and I have even personally done some fluid simulation work.
There are perfectly good models for weather in an abstract sense: Navier-Stokes plus various chemical models plus heat transfer plus radiation plus however you feel like modeling the effect of the ground and the ocean surface. (Or use Navier-Stokes for the ocean too!)
But this is wildly impractical. The Earth is too big. The relevant distance and time scales are pretty short, and the resulting grid would be too large. Not to mention that we have no way of actually measuring the whole atmosphere or even large sections of it in its full 3D glory in anything remotely close to the necessary amount of detail.
Go read the Wikipedia article, and contemplate the "Computation" and "Parameterization" sections. This works, but it's horrible. It's doing something akin to making an effective theory (the model actually solved) out of a larger theory (Navier-Stokes+), but we can't even measure the fields in the effective theory. We might want to model a handful of fields at 0.25 degrees (of lat/long) resolution, but we're getting the data from a detailed vertical slice every time someone launches a weather balloon. Which happens quite frequently, but not continuously and not at 0.25 degree spatial increments.
Hence my point: Google's model is sort of learning an effective theory instead of developing one from first principles based on the laws of physics and chemistry.
edit: I once worked in a fluid dynamics lab on something that was a bit analogous. My part of the lab was characterizing actual experiments (burning liquids and mixing of gas jets). Another group was trying to simulate related systems on supercomputers. (This was a while ago. The supercomputers were not very capable by modern standards.)
The simulation side used a 3D grid fine enough (hopefully) to capture the relevant dynamics but not so fine that the simulation would never finish. Meanwhile, we measured everything in 1D 2D! We took pictures and videos with cameras at various wavelengths. We injected things into the fluids for better visualization. We measured the actual velocity at one location (with decent temporal resolution) and hoped our instrumentation for that didn’t mess up the experiment too much. We tried to arrange to know the pressure field in the experiment by setting it up right.
With the goal of understanding the phenomena, I think this was the right approach. But if we just wanted to predict future frames of video from past frames, I would expect a nice ML model to work better. (Well, I would expect it to work better now. The state of the art was not so great at the time.)
Weather models are routinely run at resolutions as fine as 1-3 km - fine enough that we do not parameterize things like convection and allow the model to resolve these motions on its native grid. We typically do this over limited areas (e.g. domain the size of a continent), but plenty of groups have such simulations globally. It's just not practical (cost for compute and resulting data) to do this regularly, and it offers little by way of direct improvement in forecast quality.
Furthermore, we don't have to necessarily measure the whole atmosphere in 3D; physical constraints arising from Navier-Stokes still apply, and we use them in conjunction with the data we _do_ have to estimate a full 3D atmospheric state complete with uncertainties.
(If someone with knowledge or experience can chime in, please feel free.)
To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
If you're talking about small scale phenomena (less than 1km), then this wouldn't help other than to be able to signal when the conditions are such that these phenomena are more likely to happen.
Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
The API is unusably slow, the only way is to use the AWS, GCP or Azure mirrors, but they miss a lot of variables and are updated sparingly or with a delay.
It says in the article that it runs on Google's tensor units. So, go down to your nearest Google data center, dodge security, and grab one. Then escape the cops.
So for a daily user, to make it a practical usage, let's say if I have a local measurement of X, I can predict, let's say, 10 days later, or even just tomorrow, or the day after tomorrow, let's say the wind direction, is it possible to do that?
If it is possible, then I will try using the sensor to measure my velocity at some place where I live, and I can run the model and see how the results look like. I don't know if it's going to accurately predict the future or within a 10% error bar range.
See for instance the pytorch geometric [1] package, which is the main implementation in pytorch. They also link to some papers there that might explain you more.
It doesnt predict rainfall so i doubt most of us will actually care about it until then. Still it depends on input data (the current state of weather etc). How are we supposed to accurately model the weather at every point in the world? Especially when tech bro Joe living in San Fran expects things to be accurate to a meter within his doorstep
It will get adopted, eventually we will have more accurate weather forecasts. Thats good for anything that depends on weather - e.g. energy consumption and production, transportation costs...
Does anybody know if its possible to initialize the model using GFS initial conditions used for the GFS HRES model? If so, where can I find this file and how can I use it? Any help would be greatly appreciated!
You can try, but other models in this class have struggled when initialized using model states pulled from other analysis systems.
ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration is stable or not - models like PanguWx do not work off-the-shelf with this approach.
Thank you for your response. Are these ML models initialized by gridded initial conditions measurements (such as the GDAS pointed out) or by NWP model forecast results (such as hour-zero forecast from the GFS)? Or are those one and the same?
Practically speaking yes. You'd not likely build a statistical model when you could build a good simulation of the underlying process if the simulation was already really fast and accurate.
Curious. How can AI/ML perform on a problem that is, as far as I understand, inherently chaotic / unpredictable ? It sounds like a fundamental contradiction to me.
Weather isn’t fundamentally unpredictable. We predict weather with a fairly high degree of accuracy (for most practical uses), and the accuracy getting better all the time.
I'm kinda surprised that this government science website doesn't seem to link sources. I'd like to read the research to understand how they're measuring the accuracy.
IMO a chaotic system will not allow for long-term forecast, but if there is any type of pattern to recognize (and I would assume there are plenty), an AI/ML model should be able to create short-term prediction with high accuracy.
To be clear: With short-term I meant the mentioned 6 hours of the article. They use those 6 hours to create forecasts for up to 10 days. I would think that the initial predictors for a phenomenon (like a hurricane) are well inside that timespan. With long-term, I meant way beyond a 14-day window.
The issue with chaotic systems is not data, is that the error grows superlinearly with time, and since you always start with some kind of error (normally due to measurement limitations) this means that after a certain time horizon the error becomes to significant to trust the prediction. That hasn't a lot to do with data quality for ML models
That’s an issue with data: If your initial conditions are wrong (Aka your data collection has any error or isn’t thorough enough) then you get a completely different result.
Every measurement has inherent errors in it - and those errors are large if the task is to measure the location and velocity of every molecule in the atmosphere.
You also need to measure the exact amount of solar radiation before it hits these molecules (which is impossible, so we assume this is constant depending on latitude and time)
These errors compound (the butterfly effect) which is why we can't get perfect predictions.
This is a limit inherent in physical systems because of physics, not really a data problem.
Because there are tons of parts of weather where chaos isn't the limiting factor currently.
There are a limited number of weather stations producing measurements, and a limited "cell size" for being able to calculate forecasts quickly enough, and geographical factors that aren't perfectly accounted for in models.
AI is able to help substantially with all of these -- from interpolation to computational complexity to geography effects.
308 comments
[ 1.9 ms ] story [ 301 ms ] threadIn any case, GraphCast is a 10-day global model, whereas MetNet is a 24-hour regional model, among other differences.
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
Furthermore, a predictive model is not working with a complete picture of the weather, but rather some limited-resolution measurements. So, even ignoring non-weather, there may be local weather phenomena detected at time t0, escaping detection at time t1, but still affecting weather at time t2.
Initializing directly from, say, geostationary and LEO satellite data with complementary surface station observations - skipping the assimilation step entirely - is clearly where this revolution is headed, but it's very important to explicitly note that we're not there yet (even in a research capacity).
Weather forecasting has been moving focus towards ensembles to account for uncertainty in forecasts. I see a future of large ensembles of ML models being ran hourly incorporating the latest measurements
But it is exciting that they are able to recognize patterns in multi year and produce medium term forecasts.
Some comments here suggest this replaces supercomputers models. This would a wrong conclusion.It does not (the paper explicitly states this). It uses their output as input data.
We have dozens of complementary and contradictory sources of weather information. Different types of satellites measuring EM radiation in different bands, weather stations, terrestrial weather radars, buoys, weather balloons... it's a massive hodge-podge of different systems measuring different things in an uncoordinated fashion.
Today, it's not really practical to assemble that data and directly feed it into an AI system. So the state-of-the-art in AI weather forecasting involves using an intermediate representation - "reanalysis" datasets which apply a sophisticated physics based weather model to assimilate all of these data sets into a single, self-consistent 3D and time-varying record of the state of the atmosphere. This data is the unsung hero of the weather revolution - just as the WMO's coordinated synoptic time observations for weather balloons catalyzed effective early numerical weather prediction in the 50's and 60's, accessible re-analysis data - and the computational tools and platforms to actually work with these peta-scale datasets - has catalyzed the advent of "pure AI" weather forecasting systems.
I always have a lot of ideas about using AI to solve very small scale weather forecasting issues, but there's just so much to it. It's always a learning experience for sure.
https://www.ecmwf.int/en/forecasts/documentation-and-support
Another very cool application could incorporate generative modeling. Inject a bit of uncertainty in a some observations and study how the manifold of forecast outputs changes... ultimately, you could tackle things like studying the sensitivity of forecast uncertainty for, say, a tropical cyclone or nor'easter relative to targeted observations. Imagine a tool where you could optimize where a Global Hawk should drop rawindsondes over the Pacific Ocean to maximally decrease forecast uncertainty for a big winter storm impacting New England...
We may not be able to engineer the weather anytime soon, but in the next few years we may have a new type of crystal ball for anticipating its nuances with far more fidelity than ever before.
You could numerically render a 4k scene at 120FPS at extreme cost, or you could render a 2k scene at 60FPS, then feed that to DLSS to get a close-enough approximation of the former at enormous energy and hardware savings.
The global models tend to consistently miss in places that have local weather "quirks" - which is why local forecasters tend to do better than, say, accuweather, where it just posts what the models say.
Local forecasters might have learned over time that, in early Autumn, the models tend to overpredict rain, and so when they give their forecasts, they'll tweak the predictions based on the model tendencies.
National weather institutions sometimes do this, since they don't have the resources to run a massive supercomputer model.
So there's radial-coordinate voxels that represent a low resolution of the physical state of the entire atmosphere.
> ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute resources and higher resolution data.
I can't wait to see how far other people take this.
It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
Building a statistical model to approximate a physical process isn't a new idea for sure.. there are literally dozens of them for weather.. the idea itself isn't really even iterative, it's the same idea... but it's all in the execution. If you built a model to predict stock prices tomorrow and it generated 1000% pa, it wouldn't be reasonable for me to call it iterative.
I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs:
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
https://storage.googleapis.com/deepmind-media/DeepMind.com/B...
Dask-jobqueue https://jobqueue.dask.org/ :
> provides cluster managers for PBS, SLURM, LSF, SGE and other [HPC supercomputer] resource managers
Helpful tools for this work: Dask-labextension, DaskML, CuPY, SymPy's lambdify(), Parquet, Arrow
GFS: Global Forecast System: https://en.wikipedia.org/wiki/Global_Forecast_System
TIL about Raspberry-NOAA and pywws in researching and summarizing for a comment on "Nrsc5: Receive NRSC-5 digital radio stations using an RTL-SDR dongle" (2023) https://news.ycombinator.com/item?id=38158091
I don't using raw historical data would work for any data intensive model - afaik the data is patchy - there are spots where we don't have that many datapoints - e.g. middle of ocean... Also there are new satelites that are only available for the last x years and you want to be able to use these for the new models. So you need a re-analysis of what it would look like if you had that data 40 years ago...
Also its very convinient dataset because many other models trained on it: https://github.com/google-research/weatherbench2 so easy to do benchmarking..
Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.
See: https://open-meteo.com
It was super easy and the responses are very fast.
Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.
https://www.reuters.com/graphics/CLIMATE-CHANGE-ICE-SHIPLOGS...
KML files for storm tracks are still the best way to go. You could calculate storm tracks yourself for other weather models like DWD ICON, ECMWF IFS or MeteoFrance ARPEGE, but storm tracks based on GFS ensembles are easy to use with sufficient accuracy
Does anyone have a compare this API with the latest API we have here?
With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties.
Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.
So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"
https://www.nhc.noaa.gov/verification/verify5.shtml
Our 96 hour projections are as accurate today as the 24 hour projections were in 1990.
They're very cautious about naming a "best" model though!
> Weather forecasting is a multi-faceted problem with a variety of use cases. No single metric fits all those use cases. Therefore,it is important to look at a number of different metrics and consider how the forecast will be applied.
The amount of brides I've had to calm down when rain was forecast for their day is pretty high. In my experience, in my region, precipitation forecasts more than 3 days out are worthless except for when it's supposed to rain for several days straight. Temperature/wind is better but it can still swing one way or the other significantly.
For other types of shoots I'd tell people that ideally we'd postpone on the day of, and only to start worrying about it the day before the shoot.
I'm in Minnesota, so our weather is quite a bit more dynamic than many regions, for what it's worth.
Since Open-Meteo continuously downloads every weather model run, the resulting time series closely resembles assimilated gridded data. GraphCast relies on the same data to initialize each weather model run. By comparing past forecasts to future assimilated data, we can assess how much a weather model deviates from the "truth," eliminating the need for weather station data for comparison. This same principle is also applied to validate GraphCast.
Moreover, storing past weather model runs can enhance forecasts. For instance, if a weather model consistently predicts high temperatures for a specific large-scale weather pattern, a machine learning model (or a simple multilinear regression) can be trained to mitigate such biases. This improvement can be done for a single location with minimal computational effort.
Based on historical data!
I'm used to working with different weather stations, e.g. seeing different snowfall prediction at the bottom of a mountain, halfway up, and at the top, where the coordinates are quite similar.
I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube.
Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme.
While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours.
You can find more information here: https://openmeteo.substack.com/i/64601201/how-data-are-store...
I just hit the daily limit on the second request at https://climate-api.open-meteo.com/v1/climate
I see the limit for non-commercial use should be "less than 10.000 daily API calls". Technically 2 is less than 10.000, I know, but still I decided to drop you a comment. :)
or 1 request every ~9 seconds.
Maybe you just didn't space them enough.
It's a pleasure being able to use it in https://weathergraph.app
Enjoy the data directly from the source producing them.
American weather agency: https://www.nco.ncep.noaa.gov/pmb/products/gfs/
European weather agency: https://www.ecmwf.int/en/forecasts/datasets/open-data
The data’s not necessarily east to work with, but it’s all there, and you get all the forecast ensembles (potential forecasted weather paths) too
On top of this, surely the actual observations that feed into the model are terrible -- they come from weather stations, sounding rockets, balloons, radar, etc, none of which seem likely to be especially accurate in all locations. Except that, where a weather station exists, the output of that station is the observation that people care about -- unless you're in an airplane, you don't personally care about the geopotential, but you do care about how windy it is, what the temperature and humidity are, and how much precipitation there is.
ISTM these dynamics ought to be better captured by learning them from actual observations than from trying to map physics both ways onto the rather limited datasets that are available. And a trained model could also learn about the idiosyncrasies of the observation and the extra bits of forcing (buildings, etc) that simply are not captured by the inputs.
(Heck, my personal in-my-head neural network can learn a mapping from NWS forecasts to NWS observations later in the same day that seems better than what the NWS itself produces. Surely someone could train a very simple model that takes NWS forecasts as inputs and produces its estimates of NWS observations during the forecast period as outputs, thus handling things like "the NWS consistently underestimates the daily high temperature at such-and-such location during a summer heat wave.")
There is not what I would call a 3D spherical coordinate system. There’s no field f defined as f(theta, phi, r) — ther are 48 fields that are functions of theta and phi.
It also seems like some of your facts differ from theirs, may I ask how far you read into the paper?
There are perfectly good models for weather in an abstract sense: Navier-Stokes plus various chemical models plus heat transfer plus radiation plus however you feel like modeling the effect of the ground and the ocean surface. (Or use Navier-Stokes for the ocean too!)
But this is wildly impractical. The Earth is too big. The relevant distance and time scales are pretty short, and the resulting grid would be too large. Not to mention that we have no way of actually measuring the whole atmosphere or even large sections of it in its full 3D glory in anything remotely close to the necessary amount of detail.
Go read the Wikipedia article, and contemplate the "Computation" and "Parameterization" sections. This works, but it's horrible. It's doing something akin to making an effective theory (the model actually solved) out of a larger theory (Navier-Stokes+), but we can't even measure the fields in the effective theory. We might want to model a handful of fields at 0.25 degrees (of lat/long) resolution, but we're getting the data from a detailed vertical slice every time someone launches a weather balloon. Which happens quite frequently, but not continuously and not at 0.25 degree spatial increments.
Hence my point: Google's model is sort of learning an effective theory instead of developing one from first principles based on the laws of physics and chemistry.
edit: I once worked in a fluid dynamics lab on something that was a bit analogous. My part of the lab was characterizing actual experiments (burning liquids and mixing of gas jets). Another group was trying to simulate related systems on supercomputers. (This was a while ago. The supercomputers were not very capable by modern standards.)
The simulation side used a 3D grid fine enough (hopefully) to capture the relevant dynamics but not so fine that the simulation would never finish. Meanwhile, we measured everything in 1D 2D! We took pictures and videos with cameras at various wavelengths. We injected things into the fluids for better visualization. We measured the actual velocity at one location (with decent temporal resolution) and hoped our instrumentation for that didn’t mess up the experiment too much. We tried to arrange to know the pressure field in the experiment by setting it up right.
With the goal of understanding the phenomena, I think this was the right approach. But if we just wanted to predict future frames of video from past frames, I would expect a nice ML model to work better. (Well, I would expect it to work better now. The state of the art was not so great at the time.)
Furthermore, we don't have to necessarily measure the whole atmosphere in 3D; physical constraints arising from Navier-Stokes still apply, and we use them in conjunction with the data we _do_ have to estimate a full 3D atmospheric state complete with uncertainties.
To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
I think that only some variables from the HRES are free, but not 100% sure.
Any pointers?
To use the data in live fashion I think you would need to get license from ECMWF...
If it is possible, then I will try using the sensor to measure my velocity at some place where I live, and I can run the model and see how the results look like. I don't know if it's going to accurately predict the future or within a 10% error bar range.
Any way to run this at even higher resolution, like 1 km? Could this resolve terrain forced effects like lenticular clouds on mountain tops?
Why? Other AI studios seem to work on gimmicks while DeepMind seems to work on genuinely useful AI applications [0].
Thanks for the good work!
[0] Not to say that Chat GPT & Midjourney are not useful, I just find DeepMinds quality of research more interesting.
Does GraphCast come close to them?
[1] https://pytorch-geometric.readthedocs.io/en/latest/
ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration is stable or not - models like PanguWx do not work off-the-shelf with this approach.
[1]: https://github.com/ecmwf-lab/ai-models
https://scijinks.gov/forecast-reliability
You also need to measure the exact amount of solar radiation before it hits these molecules (which is impossible, so we assume this is constant depending on latitude and time)
These errors compound (the butterfly effect) which is why we can't get perfect predictions.
This is a limit inherent in physical systems because of physics, not really a data problem.
Edit: I do see a benefit to the idea if you compare it to the Chaos Theorists “gaining intuition” about systems.
There are a limited number of weather stations producing measurements, and a limited "cell size" for being able to calculate forecasts quickly enough, and geographical factors that aren't perfectly accounted for in models.
AI is able to help substantially with all of these -- from interpolation to computational complexity to geography effects.