Stabilizing storage is high priority for us! You won’t notice anything new while using DuckDB, but we’ve actually rewritten our entire serialization framework in this release. This was a big step towards stable storage
We plan to update our results on ClickBench at some point, as we’ve made some improvements there in this release. I’d expect performance to be similar
This is very true. DuckDB does not support JSON because it’s a good tabular format, but because JSON is ubiquitous, and there are many use cases where querying JSON dumps for analytics is useful.
DuckDB will use multiple threads for reading the same file. Each thread will read different parts of the file, but the output will be in the order that the file came in due to DuckDB’s order preserving parallelism.
There are plenty of TopN sorts in TPC-DS, but not many (if any) regular sorts, so no. There are order-dependent window functions in there though, which did show a speed up.
You are right, we could probably re-run SQLite purely in memory, but only because macos dynamically allocates additional swap. However, I would not expect much improved performance, because I do not believe that SQLite…
Exactly. Even when it’s inlined, having a comparator with if/else is much, much slower than a single memcmp
Customer fits in memory, whereas catalog_sales does not. We chose to remove SQLite from the results because it was so much slower. The plots are much less readable when they are stretched out by something that is slower…
I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager. The M1 + SSD performs really well here. We tried to an external…
We wanted to use the same setup for all experiments, so we had to choose for an on-disk DB for SQLite, because TPC-DS SF100 catalog_sales does not fit in 16GB memory.
Stabilizing storage is high priority for us! You won’t notice anything new while using DuckDB, but we’ve actually rewritten our entire serialization framework in this release. This was a big step towards stable storage
We plan to update our results on ClickBench at some point, as we’ve made some improvements there in this release. I’d expect performance to be similar
This is very true. DuckDB does not support JSON because it’s a good tabular format, but because JSON is ubiquitous, and there are many use cases where querying JSON dumps for analytics is useful.
DuckDB will use multiple threads for reading the same file. Each thread will read different parts of the file, but the output will be in the order that the file came in due to DuckDB’s order preserving parallelism.
There are plenty of TopN sorts in TPC-DS, but not many (if any) regular sorts, so no. There are order-dependent window functions in there though, which did show a speed up.
You are right, we could probably re-run SQLite purely in memory, but only because macos dynamically allocates additional swap. However, I would not expect much improved performance, because I do not believe that SQLite…
Exactly. Even when it’s inlined, having a comparator with if/else is much, much slower than a single memcmp
Customer fits in memory, whereas catalog_sales does not. We chose to remove SQLite from the results because it was so much slower. The plots are much less readable when they are stretched out by something that is slower…
I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager. The M1 + SSD performs really well here. We tried to an external…
We wanted to use the same setup for all experiments, so we had to choose for an on-disk DB for SQLite, because TPC-DS SF100 catalog_sales does not fit in 16GB memory.