I’m a big fan of tmducken. We use it heavily in our prod systems. That said, we’ve recently started exploring ducktape [1] in our new projects and have been really impressed with the performance. It also support more complex types on insertions and queries which has been helpful for us. Not affiliated with the project, but just wanted to show it some love since it’s a bit newer. It was created by an active contributor to tmducken.
At Cronitor we use ClickHouse, but we're leaving it behind for our next product and building directly on Parquet and DuckDB.
We think the future of observability in the AI age is self-hosted directly on NVMe backed by cheap and limitless object storage. I don't want to send customer conversations and agent thoughts to a giant multi-tenant borg SaaS database like Sentry or BetterStack.
Duckdb CLI is a powerhouse, it can load files as diverse as gzipped json lines, so you can stuff compressed logs straight into a directory yet still easily query them with SQL when you need to.
> Developing such a high quality power tool in such an open manner is honorable.
Credit where credit is due, I would say their efforts are more than just "honorable", I could easily prefix that with an "extremely" and maybe add in a ", most excellent" afterwards.
Impressive, you can really do a lot on a single node when it comes to big-data queries nowadays, I agree too many jump straight to a Spark cluster or something similar when you can just write a small script on a single node.
> but it's irritating to perform a full row-to-column conversion through an inefficient, non-batched API in order to get the data through JDBC and into TMD
[...]
> In the last two years, DuckDB improved a lot. Importantly, the C interface now provides a batched system for both inserts and querying, which enables processing very large joins
That's why. Basically JDBC doesn't offer the batched APIs needed to get the same performance.
DuckDB, for those who don't know it, has a great extension system, for example it can directly use OLTP databases such as PostgreSQL, MySQL, SQLite, SQL Server; cloud data warehouses/data lakes and big data formats (Iceberg, Delta, Snowflake, Hive, ORC, Parquet, AVRO), other data sources (ODBC), storage (S3), and much much more.
It will certainly allow you to automate tons of stuff, such as data movement between those various sources, data transformations (ETL/ELT) and even simple data conversions - such as from CSV to Parquet, or from ORC to JSON etc. etc. Happy quacking!
14 comments
[ 0.23 ms ] story [ 15.3 ms ] thread[1] https://github.com/dynamic-alpha/ducktape
We think the future of observability in the AI age is self-hosted directly on NVMe backed by cheap and limitless object storage. I don't want to send customer conversations and agent thoughts to a giant multi-tenant borg SaaS database like Sentry or BetterStack.
Credit where credit is due, I would say their efforts are more than just "honorable", I could easily prefix that with an "extremely" and maybe add in a ", most excellent" afterwards.
most of my query now are llm generated. i created custom api to connect llm to any databases im currently using.
I did consider tmducken in the beginning.
(Disclaimer: I maintain o11ylite)
[...]
> In the last two years, DuckDB improved a lot. Importantly, the C interface now provides a batched system for both inserts and querying, which enables processing very large joins
That's why. Basically JDBC doesn't offer the batched APIs needed to get the same performance.
https://duckdb.org/docs/current/core_extensions/overview
https://duckdb.org/community_extensions/list_of_extensions