Any thought about a pass-through server for various non-local databases? It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL). For others it might just be a matter of mapping…
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui `duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your…
duckdb -ui pre_seeded_db.db duckdb -ui -init startup.sql where startup.sql contains various statements/commands like `.open pre_seeded_db.db` Alternatively place statements/commands in `~/.duckdbrc` and just run `duckdb…
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical. It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even…
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions - Push dynamically generated join filters through UNION, UNNEST and AGGREGATE - Fix arrow table filters - [Arrow]…
Perhaps I don't fully understand what you're saying CREATE TABLE person (name VARCHAR, age BIGINT); INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25); CREATE OR REPLACE MACRO…
For read-oriented interactions... ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources or SELECT * FROM <relation> WHERE ... where the "relation" is a collection of Arrow stream from a…
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible. My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other…
And all the Arrow parts work together quite nicely. ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ? The result highlights your exact point: why take your data and transpose it twice? It's quite an exciting…
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options. https://github.com/tobymao/sqlglot
Maybe something like this? https://duckdb.org/2024/10/02/pyodide.html
Probably also worth mentioning that DuckDB can interact with SQLite dbs. https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h... Thus potentially making duckdb an…
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising. https://motherduck.com/blog/differential-storage-building-bl... Hopefully it finds its way into duckdb's repo some day.
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables") https://duckdb.org/docs/sql/query_syntax/with#cte-materializ... Obviously, the materialization…
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation. https://duckdb.org/docs/api/python/spark_api And while on the subject of…
Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing…
Real awesome project. This would be even better if some of the differential storage functionality makes its way into duckdb. https://motherduck.com/blog/differential-storage-building-bl... Unsure if it's even possible,…
Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.
It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you…
Yes, DuckDB does do Iceberg. https://duckdb.org/docs/extensions/iceberg
Interested to hear why you switched from DataFusion?
Via the AWS extension https://duckdb.org/docs/guides/network_cloud_storage/s3_impo... https://duckdb.org/docs/extensions/aws
DuckDB can do some pushdowns to certain file formats like parquet, but every release seems to be getting better and better at doing it. Parquet pushdowns combined with Hive structuring is a pretty good combination.…
DuckDB? https://duckdb.org/2024/03/29/external-aggregation.html https://duckdb.org/2021/06/25/querying-parquet.html If your DB is mysql or postgres, then you could read a stream from parquet, transform inline and write…
In my experience, it's always come down to either using a specific database type for the wrong job, or using a database type the wrong way. - Treating an OLAP like an OLTP. - Using a document store for relational data…
Any thought about a pass-through server for various non-local databases? It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL). For others it might just be a matter of mapping…
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui `duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your…
duckdb -ui pre_seeded_db.db duckdb -ui -init startup.sql where startup.sql contains various statements/commands like `.open pre_seeded_db.db` Alternatively place statements/commands in `~/.duckdbrc` and just run `duckdb…
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical. It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even…
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions - Push dynamically generated join filters through UNION, UNNEST and AGGREGATE - Fix arrow table filters - [Arrow]…
Perhaps I don't fully understand what you're saying CREATE TABLE person (name VARCHAR, age BIGINT); INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25); CREATE OR REPLACE MACRO…
For read-oriented interactions... ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources or SELECT * FROM <relation> WHERE ... where the "relation" is a collection of Arrow stream from a…
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible. My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other…
And all the Arrow parts work together quite nicely. ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ? The result highlights your exact point: why take your data and transpose it twice? It's quite an exciting…
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options. https://github.com/tobymao/sqlglot
Maybe something like this? https://duckdb.org/2024/10/02/pyodide.html
Probably also worth mentioning that DuckDB can interact with SQLite dbs. https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h... Thus potentially making duckdb an…
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising. https://motherduck.com/blog/differential-storage-building-bl... Hopefully it finds its way into duckdb's repo some day.
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables") https://duckdb.org/docs/sql/query_syntax/with#cte-materializ... Obviously, the materialization…
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation. https://duckdb.org/docs/api/python/spark_api And while on the subject of…
Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing…
Real awesome project. This would be even better if some of the differential storage functionality makes its way into duckdb. https://motherduck.com/blog/differential-storage-building-bl... Unsure if it's even possible,…
Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.
It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you…
Yes, DuckDB does do Iceberg. https://duckdb.org/docs/extensions/iceberg
Interested to hear why you switched from DataFusion?
Via the AWS extension https://duckdb.org/docs/guides/network_cloud_storage/s3_impo... https://duckdb.org/docs/extensions/aws
DuckDB can do some pushdowns to certain file formats like parquet, but every release seems to be getting better and better at doing it. Parquet pushdowns combined with Hive structuring is a pretty good combination.…
DuckDB? https://duckdb.org/2024/03/29/external-aggregation.html https://duckdb.org/2021/06/25/querying-parquet.html If your DB is mysql or postgres, then you could read a stream from parquet, transform inline and write…
In my experience, it's always come down to either using a specific database type for the wrong job, or using a database type the wrong way. - Treating an OLAP like an OLTP. - Using a document store for relational data…