The last year of DuckDB enhancements feel like the shift from in-process execution engine (which it is phenomenal at) to an engine that can serve as the foundation of a cloud data warehouse. I know the founders were reticent about not wanting to build that, but I have a feeling it is in the works.
I <3 DuckDB. It has become one of my go to tools for storing, data processing , integrations and now even graph. More importantly it's fun to use because it is so portable. Looking forward to v2.
Not OP, but for me, the lack of essentially any type system in SQLite makes it a total no-go for storing data long-term or that more than one application needs to access. Date/time being an especially painful footgun in SQLite.
I view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.
DuckDB is heavily focused on analytical workflows, so it stores data in a compressed columnar format. So "rewrite one row" is more efficient in SQLite, but "sum all the values in this column" is vastly more efficient in DuckDB.
That's the core of the difference, to me. But the projects also have a pretty different ethos. SQLite has always prioritized simplicity, often at the expense of features, whereas DuckDB's approach to features seems to be "the more the merrier." E.g. the star feature of this new release, client/server mode. Plenty of people would love to see something similar in SQLite, but it doesn't seem likely.
What do you plan on stripping and what's your target? The Emscripten based build is ~10Mb. I have a component build so I'd be interesting on how you'd like to break it up.
DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable.
DuckDB out of the box may not be great. But you have DuckLake, Quack, and even DeepSeek made their own distributed DB based on DuckDB: https://github.com/deepseek-ai/smallpond
I don't think DuckDB itself can coordinate work across multiple nodes. But you could put it behind an HTTP layer and scale horizontally based on resource utilization?
Looks like an awesome release, but the smell of AI from that post is horrid.
Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
I don't really get an AI smell on this, in fact I see multiple parts that an AI would have corrected - grammatical issues, personal writing quirks, etc.
Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
I’m currently contemplating that MySQL apparently cannot do an INSTANT change of the collation of an unindexed column, even though, AFAICT, it has no effect whatsoever on the on-disk format or any data structure at all except for the metadata saying what the column type is.
I do not enjoy dealing with text encodings and collations in databases.
If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or you need to rely on full table scans or at least scans of big blocks. DuckDB does the latter really well, but I think it would be quite nice for some workflows to have explicit ordering. Also, I bet compression could work a lot better with ordering hints.
All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.
[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.
Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a giant several hundred step dbt pipeline" to "query the output of said pipeline" to "read/query a csv on disk" with the exact same tool is just so nice. If I could centrally manage said asset more akin to a traditional database, I'd be very happy.
I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I'd really like way too many things about duckdb to replace it.
I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop), but there's so much I like about it, and it's really nice that each tenant can have total separation with custom schemas and that it's straightforward to pile data into object storage and form your own lake.
And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ?
The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine).
But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).
With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".
DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.
i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it
is your setup running into problems that makes you need something more?
Clickhouse, as I mentioned, can be a good final layer, as can postgres.
You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.
duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).
All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.
For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.
Similar. Noticed DuckDB ever since an old article 'what db should I use' for local small data warehousing. The author was blown away that DuckDB seemed super naturally quick. It was I think columnar store + compression facilitated that? It made duckdb load compressed + on the fly decompress = faster than even reading the uncompressed data. Had forgotten most of it. But was used to mmap-ed files + columnar storing of Kdb. Was pleased that the author was clued to notice the power of that.
Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)
I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.
Really looking forward to that new Async system, especially when reading/querying against thousands of parquet files. This is going to monumentally affect me and my work - I have to query against millions of massive parquet files and the speed has already been rather wonderful, but if those metrics are to be even 100% in range, this is going to make life so much better.
Disappointed, since I was expecting they would rewrite the implementation from C++ to Zig. I bet that would increase the number of positive pull requests they get, since most developers prefer to stay away from C++ nowadays.
I've been working on a demo database project, and have been really impressed by the UI. So glad they decided to put more effort into it, it has made building a "follow along" tutorial really nice.
Are there improvements in how memory_limit works? I often had DuckDB get OOM killed because it went beyond its limit. It's definitely one of the reasons why I usually have an AI tune the environment for my datasets.
I look forward to DuckDB being the engine that underpins the next gen of analytical data tooling. DuckLake already looks amazing and with the Quack protocol seems like it will be a great natural fit for lots more types of tooling, such as sensor data etc.
which statistical functions do you want? i’m curious because i love duckdb and use it for a variety of projects but always want to learn more about how to use tools better.
Basic regression and interpolation for starters; perhaps some classification and outlier detection too. Disclaimer: I self-studied applied DuckDB for my job, I don't know if these already exist.
113 comments
[ 0.23 ms ] story [ 37.3 ms ] threadI view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.
That's the core of the difference, to me. But the projects also have a pretty different ethos. SQLite has always prioritized simplicity, often at the expense of features, whereas DuckDB's approach to features seems to be "the more the merrier." E.g. the star feature of this new release, client/server mode. Plenty of people would love to see something similar in SQLite, but it doesn't seem likely.
What about the runtime size? I care this because I intend to run a stripped WASM version of DuckDB in browser.
Thanks to the team for everything!
"If we use this we get indefinite RAM indefinite CPU and do not need to host a server".
I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?
Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
For me, this sentence pops out like a sore thumb
I do not enjoy dealing with text encodings and collations in databases.
All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.
[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
https://github.com/duckdb/duckdb/discussions/8444
I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.
I made a dry run extension a few months ago (https://github.com/aleda145/duckdb-dryrun), will be so nice to build it just once and know that it will always work.
Also urge anyone to make an extension, the template makes it quite smooth: https://github.com/duckdb/extension-template
I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I'd really like way too many things about duckdb to replace it.
[1]: If you care: https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spati...
And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).
With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.
i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it
is your setup running into problems that makes you need something more?
Clickhouse, as I mentioned, can be a good final layer, as can postgres.
You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.
duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).
All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.
For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.
Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)
Latter I read this https://openai.com/index/inside-our-in-house-data-agent/ and thought "but of course".
I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.
DuckDB is seriously an incredible utility.
https://duckdb.org/community_extensions/extensions/stats_duc...
https://duckdb.org/community_extensions/extensions/stochasti...
DuckDB 1.0 was in 2024. DuckDB 2.0 (new API, new storage format, new ...) is in 2026.
SQLite has been 3.x since 2004.