6 comments of 11

[ 4.0 ms ] story [ 13.6 ms ] thread
there's some great stuff in here, particularly the direct-to-binary codegen, threads instead of processes and sync batching. you must have some intuitive notion about where the performance gains come from.
Would there be an advantage to reusing SIMD-optimized libraries for this?

Polars Rust is built on Arrow and packed_simd.

pola-rs/polars: https://github.com/pola-rs/polars

polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd

lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance

Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/

substrait's portable query plans: https://substrait.io/ , https://github.com/ibis-project/ibis-substrait

Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244

cargo-fuzz, TLA+

You should submit it to ClickBench for verification.