> It would be rather misguided not to build a database protocol on top of HTTP in 2026
This is wrong, HTTP is bad for transferring large amount of data and it is also bad for doing streaming.
It is bad for large amount of data because you have timeout issues on some clients, you hit request/response size limits etc.
It is obviously bad for streaming as there is no concept of streaming in it.
It is comical to go the path of least resistance so lazy people can put a reverse proxy on top of it. And then say HTTP is the only relevant way to do it in 2026.
The benchmark doesn't seem to mean much as TCP can max out 50GB/s on a single thread. Pretty sure it can do more than that even. So you could be using anything that isn't terrible and you should get max performance out of this.
Also the protocol is something else from the format. For example if you are transferring mp4 over ftp and http you can compare that.
If you are transferring different things over different protocols then the comparison means nothing.
The benchmark graph for bulk transfer should show more granularity so it is possible to understand how much of the % of the hardware limit it is reaching. Similar to how BLAS GEMM routines are benchmarked based on the % of theoretical max flops of the hardware.
> 60 million rows (76 GB in CSV format!)
This reads a bit disingenuous.
It is dissappointing to see this instead of something like PostgreSQL protocol with support for a columnar format.
Although a maintainer answered you, watch the video from the blog. There's a WASM demo at the end, which is great. It also has a good explainer for those confused about the HTTP decision.
And I appreciate that the Hannes still appreciates the magic of the WASM. [And I keep hearing quark which makes me hungry for tangy creamy German yogurt]
This is rad. I've been eyeballing using DuckDB in my firm's internal app framework and this just solved the "but how do I horizontally scale this" problem. Kudos to the DuckDB folks. Love "Quack" for the protocol name, too.
I have a C++ application. Everything is in memory during execution. Saved to disk between session as XML. Works great, except that that it is strictly single user and some of my customers would love me to generalize it for multiple concurrent users reading and writing. Performance requirements are quite low - a few thousand records being updated by 2 or 3 people at a time. Would DuckDb + Quack be a good choice for this? Or are there better choices? I looked at SQLite, but I understand it doesn't operate as client server.
This is fantastic. I’ve been building an Excel-like but columnar spreadsheet app using DuckDB and had to reinvent the “client” through classic HTTP layer.
I was just wishing something like this existed last week. What timing.
I'm piping sensor readings into duckdb with a deno server, and couldn't use duckdb -ui to look over the data without shutting down the server. I had no interest in using the server to allow me to look at the contents of the db, so I was just going to live with it for now. This perfectly solves that, along with several other similar kinds of problems I've encountered with duckdb.
duckdb is my favourite technology of 2025/26. It has worked its way into so many of my workflows. It's integral to how I work with LLMs, how I store all kinds of data, analytics, data pipelines... I love it.
Been working on open-source projects involving storing and querying observability data (metrics, logs, traces) in parquet[0] and have been frustrated with the usability of Apache Iceberg … despite strongly agreeing and wanting to use an open storage format and catalog.
This makes Ducklake much more interesting for my use case, excited where this is going.
What drives such a high throughput difference between Quack and Arrow on high-volume operations ?
I'll try to search from source/Github, reply appreciated though, for example:
- when DuckDb bulk exports a table, does Quack benefit from pre-existing compression/encodings/0-copy where Arrow requires decode+re-encode ?
- the post mentions parallel reads, is the level of parallelism the same on Arrow vs Quack here ? Running the high throughput benchmark at resource saturation with increasing number of concurrent bulk-read clients would be more transparent
Very hyped for this and updates. Have been using my own workarounds for a while with own WAL things and then sort of generating snapshots which with duckdb is so cheap was simpler than really implementing concurrent writes and mutations but this will make it so much easier.
27 comments
[ 4.4 ms ] story [ 57.6 ms ] threadI can definitely see exploring this for some homelab use.
This is wrong, HTTP is bad for transferring large amount of data and it is also bad for doing streaming.
It is bad for large amount of data because you have timeout issues on some clients, you hit request/response size limits etc.
It is obviously bad for streaming as there is no concept of streaming in it.
It is comical to go the path of least resistance so lazy people can put a reverse proxy on top of it. And then say HTTP is the only relevant way to do it in 2026.
The benchmark doesn't seem to mean much as TCP can max out 50GB/s on a single thread. Pretty sure it can do more than that even. So you could be using anything that isn't terrible and you should get max performance out of this.
Also the protocol is something else from the format. For example if you are transferring mp4 over ftp and http you can compare that.
If you are transferring different things over different protocols then the comparison means nothing.
The benchmark graph for bulk transfer should show more granularity so it is possible to understand how much of the % of the hardware limit it is reaching. Similar to how BLAS GEMM routines are benchmarked based on the % of theoretical max flops of the hardware.
> 60 million rows (76 GB in CSV format!)
This reads a bit disingenuous.
It is dissappointing to see this instead of something like PostgreSQL protocol with support for a columnar format.
> Not yet, but we are working on it!
Seems like a niche use case, but it's the one I'm most interested in.
Our lakehouse uses ducklake with postgres as the catalog. Seems like a DuckDB / Quack catalog would be an excellent alternative.
And I appreciate that the Hannes still appreciates the magic of the WASM. [And I keep hearing quark which makes me hungry for tangy creamy German yogurt]
I'm piping sensor readings into duckdb with a deno server, and couldn't use duckdb -ui to look over the data without shutting down the server. I had no interest in using the server to allow me to look at the contents of the db, so I was just going to live with it for now. This perfectly solves that, along with several other similar kinds of problems I've encountered with duckdb.
duckdb is my favourite technology of 2025/26. It has worked its way into so many of my workflows. It's integral to how I work with LLMs, how I store all kinds of data, analytics, data pipelines... I love it.
This makes Ducklake much more interesting for my use case, excited where this is going.
[0] https://github.com/smithclay/duckdb-otlp
I'll try to search from source/Github, reply appreciated though, for example:
- when DuckDb bulk exports a table, does Quack benefit from pre-existing compression/encodings/0-copy where Arrow requires decode+re-encode ?
- the post mentions parallel reads, is the level of parallelism the same on Arrow vs Quack here ? Running the high throughput benchmark at resource saturation with increasing number of concurrent bulk-read clients would be more transparent
ADBC: https://github.com/gizmodata/adbc-driver-quack JDBC: https://github.com/gizmodata/quack-jdbc