Any thought about a pass-through server for various non-local databases? It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL). For others it might just be a matter of mapping…
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui `duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your…
duckdb -ui pre_seeded_db.db duckdb -ui -init startup.sql where startup.sql contains various statements/commands like `.open pre_seeded_db.db` Alternatively place statements/commands in `~/.duckdbrc` and just run `duckdb…
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical. It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even…
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions - Push dynamically generated join filters through UNION, UNNEST and AGGREGATE - Fix arrow table filters - [Arrow]…
Perhaps I don't fully understand what you're saying CREATE TABLE person (name VARCHAR, age BIGINT); INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25); CREATE OR REPLACE MACRO…
I've been eagerly awaiting this for a couple of months now. And I've long wondered why there hasn't been such an implementation/extension for Flight especially as there are extensions for mysql, postgres, sqlite... Only…
For read-oriented interactions... ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources or SELECT * FROM <relation> WHERE ... where the "relation" is a collection of Arrow stream from a…
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible. My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other…
And all the Arrow parts work together quite nicely. ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ? The result highlights your exact point: why take your data and transpose it twice? It's quite an exciting…
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options. https://github.com/tobymao/sqlglot
Maybe something like this? https://duckdb.org/2024/10/02/pyodide.html
Probably also worth mentioning that DuckDB can interact with SQLite dbs. https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h... Thus potentially making duckdb an…
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising. https://motherduck.com/blog/differential-storage-building-bl... Hopefully it finds its way into duckdb's repo some day.
Ibis looks very promising. There are also other ways to use both. https://duckdb.org/docs/guides/python/polars.html All of this dataframe compatibility is awesome. (much thanks to Arrow and others)
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables") https://duckdb.org/docs/sql/query_syntax/with#cte-materializ... Obviously, the materialization…
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation. https://duckdb.org/docs/api/python/spark_api And while on the subject of…
Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing…
Real awesome project. This would be even better if some of the differential storage functionality makes its way into duckdb. https://motherduck.com/blog/differential-storage-building-bl... Unsure if it's even possible,…
Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.
It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you…
Yes, DuckDB does do Iceberg. https://duckdb.org/docs/extensions/iceberg
Interested to hear why you switched from DataFusion?
DuckDB is really for OLAP/Columnar (analytical workloads) whereas SQLite is more for OLTP/Row-based (transactional). When you start running aggregations/windows over large amounts of data, you'll soon see the difference…
Via the AWS extension https://duckdb.org/docs/guides/network_cloud_storage/s3_impo... https://duckdb.org/docs/extensions/aws
Any thought about a pass-through server for various non-local databases? It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL). For others it might just be a matter of mapping…
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui `duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your…
duckdb -ui pre_seeded_db.db duckdb -ui -init startup.sql where startup.sql contains various statements/commands like `.open pre_seeded_db.db` Alternatively place statements/commands in `~/.duckdbrc` and just run `duckdb…
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical. It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even…
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions - Push dynamically generated join filters through UNION, UNNEST and AGGREGATE - Fix arrow table filters - [Arrow]…
Perhaps I don't fully understand what you're saying CREATE TABLE person (name VARCHAR, age BIGINT); INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25); CREATE OR REPLACE MACRO…
I've been eagerly awaiting this for a couple of months now. And I've long wondered why there hasn't been such an implementation/extension for Flight especially as there are extensions for mysql, postgres, sqlite... Only…
For read-oriented interactions... ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources or SELECT * FROM <relation> WHERE ... where the "relation" is a collection of Arrow stream from a…
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible. My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other…
And all the Arrow parts work together quite nicely. ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ? The result highlights your exact point: why take your data and transpose it twice? It's quite an exciting…
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options. https://github.com/tobymao/sqlglot
Maybe something like this? https://duckdb.org/2024/10/02/pyodide.html
Probably also worth mentioning that DuckDB can interact with SQLite dbs. https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h... Thus potentially making duckdb an…
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising. https://motherduck.com/blog/differential-storage-building-bl... Hopefully it finds its way into duckdb's repo some day.
Ibis looks very promising. There are also other ways to use both. https://duckdb.org/docs/guides/python/polars.html All of this dataframe compatibility is awesome. (much thanks to Arrow and others)
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables") https://duckdb.org/docs/sql/query_syntax/with#cte-materializ... Obviously, the materialization…
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation. https://duckdb.org/docs/api/python/spark_api And while on the subject of…
Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing…
Real awesome project. This would be even better if some of the differential storage functionality makes its way into duckdb. https://motherduck.com/blog/differential-storage-building-bl... Unsure if it's even possible,…
Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.
It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you…
Yes, DuckDB does do Iceberg. https://duckdb.org/docs/extensions/iceberg
Interested to hear why you switched from DataFusion?
DuckDB is really for OLAP/Columnar (analytical workloads) whereas SQLite is more for OLTP/Row-based (transactional). When you start running aggregations/windows over large amounts of data, you'll soon see the difference…
Via the AWS extension https://duckdb.org/docs/guides/network_cloud_storage/s3_impo... https://duckdb.org/docs/extensions/aws