rabernat
- Karma
- 13
- Created
- February 15, 2016 (10y ago)
- Submissions
- 0
Startup Founder, Scientist and Software Developer
- CEO and co-founder of Earthmover: https://earthmover.io/ - Associate professor in the Columbia University Department of Earth and Environmental Science and Lamont Doherty Earth Observatory: https://ocean-transport.github.io/ - Co-founder and community leader of the Pangeo Project: https://pangeo.io/
> It's possible but not very cost-effective to maintain separately-chunked versions of these large geospatial datasets. Like all things in tech, it's about tradeoffs. S3 storage costs about $275 TB a year. Typical…
True, but in fact, the Google ERA5 public data suffers from the exact chunking problem described in the post: it's optimized for spatial queries, not timeseries queries. I just ran a benchmark, and it took me 20 minutes…
Great post! Hi Ali! I think what's missing here is an analysis of what is gained by moving the weather data into a RDBMS. The motivation is to speed up queries. But what's the baseline? As someone very familiar with…
The Zarr format is used in some genomics workflows (see https://github.com/zarr-developers/community/issues/19) and supports a wide range of modern compressors (e.g. Zstd, Zlib, BZ2, LZMA, ZFPY, Blosc, as well as many…
PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with…
Agree 100%. This is big part of the motivation behind our new startup Earthmover: https://earthmover.io/ Our mission is to make it easier to work with scientific data at scale in the cloud, focusing mainly on the…