With a lot of weather data you end up with these large gridded files like grib, netcdf, hdf, etc. Does anyone know if theres a way to get this kind of data into a database or something where you can query it quickly? I've been wondering how people handle this kind of data for awhile.
The only idea I came up with was contouring the data into shapes and then storing those shapes in a db that can do spatial queries.
There's CDO: a collection of command line Operators to manipulate and analyse Climate and NWP model Data.
Supported data formats are GRIB 1/2, netCDF 3/4, SERVICE, EXTRA and IEG. There are more than 600 operators available:
https://code.mpimet.mpg.de/projects/cdo
> With a lot of weather data you end up with these large gridded files like grib, netcdf, hdf, etc. Does anyone know if theres a way to get this kind of data into a database or something where you can query it quickly? I've been wondering how people handle this kind of data for awhile.
GDAL has drivers for HDF4, HDF5, and netCDF.
I do a good amount of work with multispectral imagery data. My workflow for anything that doesn't immediately come as a geotiff or a jp2 is to either use gdal_translate via the command line to convert the datasets I'm interested in into a multi-band geotiff, or to open the product directly in either Python or Julia with the appropriate library.
That said, how much effort and what I do with the data depends a lot on where it comes from.
Hurricanes specifically are pretty friendly to relatively simple 2 dimensional models, since for the most part people care about the wind impacting the surface. So for that I've used raster files - xy grid is lat/lon, and you can encode a values per rgba channel, usually windspeed and central pressure, storm surge value if you have one, maybe surface friction at a point as well, I just use windspeed in these files (0). And rasters over time gives you animations, similar to this NHC product(1).
GeoJSON and Shapefiles in something like PostGIS is also viable for relatively small and simple things, like tracks or single 2D events, but once you start covering enough area or need to model in 3 dimensional grid, those more specialized file formats become more powerful.
for taking a computational fluids approach, I think yes, you're largely correct. Hurricanes have a class of models that are parametric, where you can get a 2 dimensional wind footprint based on some initial conditions; primarily central pressure, size (radius of max wind), and location. These models rely on empirical measurements to get simpler. NWS 23 (0) is an old model, it's the one I've implemented before. But there's been more refined parametric models that have come out, and updates to older ones, for example the work by Holland (1) and Wiloughby (2). Take this with a grain of salt, since I'm no expert in this field.
9 comments
[ 3.0 ms ] story [ 98.9 ms ] threadThe only idea I came up with was contouring the data into shapes and then storing those shapes in a db that can do spatial queries.
GDAL has drivers for HDF4, HDF5, and netCDF.
I do a good amount of work with multispectral imagery data. My workflow for anything that doesn't immediately come as a geotiff or a jp2 is to either use gdal_translate via the command line to convert the datasets I'm interested in into a multi-band geotiff, or to open the product directly in either Python or Julia with the appropriate library.
That said, how much effort and what I do with the data depends a lot on where it comes from.
GeoJSON and Shapefiles in something like PostGIS is also viable for relatively small and simple things, like tracks or single 2D events, but once you start covering enough area or need to model in 3 dimensional grid, those more specialized file formats become more powerful.
0: https://www.odinseye.cloud/hurricane/2016/matthew2016/ 1: https://www.nhc.noaa.gov/archive/2023/HILARY_graphics.php
I thought this is why it's impossible to simplify turbulence models by 'ignoring a dimension'?
(0) https://library.oarcloud.noaa.gov/noaa_documents.lib/NWS/TR_... (1) https://opensky.ucar.edu/islandora/object/articles%3A17282/d... (2) https://journals.ametsoc.org/view/journals/mwre/141/6/mwr-d-...