Reducing Data Lake Storage by 52% with SciPy-Style Sparse Arrays in DuckDB (blog.sturdystatistics.com)

Hi all, author here. I originally was just planning to put together a few helper functions to support sparse arrays in DuckDB, but there ended up being a lot of interesting lessons in the process of benchmarking storage and optimizing queries.

For those not wanting to read the whole story, GitHub repo with basic benchmarks and utilities is here: https://github.com/Sturdy-Statistics/duckdb-sparse-array-lis...