I think the advantage is simplicity. Why connect first to duckdb and attach the db when you can query it directly with ADBC which is guaranteed to be fast
It's more of a common database cli/shell which uses a well defined, and fast, ADBC protocol. You are basically freed from DuckDB's internal handling/runtime of query for various databases. Not to mention, this has vastly more supported databases.
With databow, the query still runs on the target database (unline duckdb), but you get one consistent CLI across different databases: connection profiles, output formats, history, scripting, and import/export behavior.
This is genuinely useful for humans (For example, I regularly juggle 6-7 different database, oltp, olap, search and key-value mixed), and even more useful for AI coding agents, because they don't have to learn and juggle a different CLI and set of flags for every database.
FWIW duckdb's optimizer does actually push some of the query down into the target database such as selects and where clauses, which you definitely want in many cases.
Cool! But as a data engineer I don't know when I would ever use this. Getting data into a centralized place so it can be joined and queried easily is like prio 1 for any data team.
I'm sure SREs will really love me doing expensive adhoc queries against production postgres /s
I've yet to work in enterprises big enough to have multi cloud data warehouses though, maybe it's more useful in that setting?
As a consultant data engineer (ish), I think it has potential. You're right that any company doing data analytics is gonna be prioritizing a single source of truth and a unified platform, but each one will choose a different set of tools, which I'll have to learn, install, and even teach, for each new client. If I can use this to both explore AND implement stuff for clients regardless of their underlying database, that would be a pretty significant win.
This is excellent! I'm not a data engineer or SRE or whatever other commenters have mentioned. But part of my job is accessing data in various formats from various places, mostly offline. This in gonna be part of my toolset and I can pipe the output into other tools like nushell too.
17 comments
[ 3.2 ms ] story [ 65.1 ms ] threadCoincidentally, I wrote an article today on how I use it for similar scenarios. It can fetch from S3, multiple databases at once, and so on.
And you get all the benefits of a database when you need to join or postprocess data from multiple sources.
https://rushter.com/blog/clickhouse-data-processing/
With databow, the query still runs on the target database (unline duckdb), but you get one consistent CLI across different databases: connection profiles, output formats, history, scripting, and import/export behavior.
This is genuinely useful for humans (For example, I regularly juggle 6-7 different database, oltp, olap, search and key-value mixed), and even more useful for AI coding agents, because they don't have to learn and juggle a different CLI and set of flags for every database.
I'm sure SREs will really love me doing expensive adhoc queries against production postgres /s
I've yet to work in enterprises big enough to have multi cloud data warehouses though, maybe it's more useful in that setting?
E.g, you don’t need a million tools to connect to the million different application databases when inspecting sources as part of setting up pipelines.
Another nice feature one would want from such a program is of course auto complete.
Reviewing the issues and PRs there provides a clue what to expect as this project matures.
Seems like a columnar version of ODBC, for OLAP instead of OLTP.
definitely a beta product, but used to use every day with redshift and snowflake