It works fine for this small set of emails, although the search isn't great, and there was more preprocessing that I would have liked. (I would prefer to be able to point a single binary at a pst or mbox file, and have it magically serve it like this, even if it means I need a VPS to serve it.)
You can point DuckDB to almost any data source and boom, you get an SQL table that you can search, sum, or join to any other data. Or you can attach existing databases from completely independent db systems, and query and join them as one, without having to first importing anything.
It feels exhilarating (if you're into that sort of thing!)
Replying to myself to add this caveat: as I found out the hard way, the limit of such a setup is that query performance depend on the performance and indexing choices of the underlying data stores.
If you have no control over those, and you don't have real time requirements, you're usually better off first importing everything into duckdb (which is usually quite fast) and doing your queries from there.
The hardest part is getting people to understand that it is interactive! People expect a document-looking webpage to be static, but we can do so much better!
DuckDB is amazing. It has completely replaced my need of using Cloudwatch or Loki etc for logs. I simply emit parquet to s3 now and use DuckDB to query locally. No HA compute or paying cloud tax for log search.
For people interested, you can do FTS in DuckDB without using the official module. I copy-pasted the relevant parts and slightly changed it for my need.
13 comments
[ 2.5 ms ] story [ 39.8 ms ] threadI'm aware of jmail.world, but they haven't (yet?) published the source code.
I had Claude hack something together recently: https://healdsburg-youcubed-emails.vercel.app/
It works fine for this small set of emails, although the search isn't great, and there was more preprocessing that I would have liked. (I would prefer to be able to point a single binary at a pst or mbox file, and have it magically serve it like this, even if it means I need a VPS to serve it.)
You can point DuckDB to almost any data source and boom, you get an SQL table that you can search, sum, or join to any other data. Or you can attach existing databases from completely independent db systems, and query and join them as one, without having to first importing anything.
It feels exhilarating (if you're into that sort of thing!)
Replying to myself to add this caveat: as I found out the hard way, the limit of such a setup is that query performance depend on the performance and indexing choices of the underlying data stores.
If you have no control over those, and you don't have real time requirements, you're usually better off first importing everything into duckdb (which is usually quite fast) and doing your queries from there.
Clickhouse seems less marketed, but seems quite similar.
https://duckdb.org/docs/current/extensions/overview#autoload...
I would love to have more detail on this mechanism.
I build a whole LLM benchmark system around it that lets you run the whole benchmark in your browser: https://sql-benchmark.nicklothian.com/#sample-queries-and-sq...
Click on a cell and you can run the SQL the LLM generated vs what the solution is: https://sql-benchmark.nicklothian.com/?highlight=ggml-org_ge...
The hardest part is getting people to understand that it is interactive! People expect a document-looking webpage to be static, but we can do so much better!
https://github.com/amr8t/blobsearch
https://github.com/nraynaud/sojourner/blob/main/PDS/db_creat...
Example: https://nraynaud.github.io/sojourner/PDS/explore.html