Show HN: DuckDB for Kafka Stream Processing (sql-flow.com)

77 points by dm03514 7mo ago ↗ HN

Hello Everyone! We built SQLFlow as a lightweight stream processing engine.

We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory!

DuckDB also supports a rich ecosystem of sinks and connectors!

https://sql-flow.com/docs/category/tutorials/

https://github.com/turbolytics/sql-flow

We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors

I would love your feedback, criticisms and/or experiences!

Thank you

5 comments

[ 3.5 ms ] story [ 21.8 ms ] thread

srameshc 7mo ago ↗

This looks brilliant, thank you. I love DuckDB and use it for lot of local data processing jobs. We have a data stream, not to the size where we need to push to BigQuery or elsewhere. I was thinking of trying something like sql-flow but I am glad now it makes the job very easy.

mbay 7mo ago ↗

I see an example with what looks like a lookup-type join against a Postgres DB. Are stream/stream joins supported, though?

The DLQ and Prometheus integration out of the box are nice.

mihevc 7mo ago ↗

How does this compare to https://github.com/Query-farm/tributary ?

itsfseven 7mo ago ↗

It would be great if this supported Pulsar too!

pulkitsh1234 7mo ago ↗

(not an expert in stream processing).. from the docs here https://sql-flow.com/docs/introduction/basics#output-sink it seems like this works on "batches" of data, how is this different from batch processing ? Where is the "stream" here ?