Show HN: Arroyo – Write SQL on streaming data (github.com)
Arroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency.
Today data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming program that constantly updates as new data comes in, providing low-latency results as soon as the data is available.
I started the Arroyo project after spending the past five years building real-time platforms at Lyft and Splunk. I saw first hand how hard it is for users to build correct, reliable pipelines on top of existing systems like Flink and Spark Streaming, and how hard those pipelines are to operate for infra teams. I saw the need for a new system that would be easy enough for any data team to adopt, built on modern foundations and with the lessons of the past decade of research and industry development.
Arroyo works by taking SQL queries and compiling them into an optimized streaming dataflow program, a distributed DAG of computation with nodes that read from sources (like Kafka), perform stateful computations, and eventually write results to sinks. That state is consistently snapshotted using a variation of the Chandy-Lamport checkpointing algorithm for fault-tolerance and to enable fast rescaling and updates of the pipelines. The entire system is easy to self-host on Kubernetes and Nomad.
See it in action here: https://www.youtube.com/watch?v=X1Nv0gQy9TA or follow the getting started guide (https://doc.arroyo.dev/getting-started) to run it locally.
34 comments
[ 0.31 ms ] story [ 85.6 ms ] threadKSQL is pretty simple and easy to run if you already have Kafka, but will be much more expensive and harder to scale due to its reliance on Kafka streams for persistence and shuffling of data in processing DAG.
And with Confluent's embrace of Flink in the past year (https://www.confluent.io/blog/cloud-kafka-meets-cloud-flink-...) it's not clear that KSQL has much of a future.
https://doc.arroyo.dev/concepts#watermarks
Also works based on fixed, pre-built pipelines. This is all very much in the style of most stream processing platforms today but I hope we’ll continue to move closer as an industry to having our cake and eating it: ingest everything in real-time, while serving any query (with joins) over the full dataset (either incrementally or ad-hoc).
Beyond that, there are systems that are more integrated end-to-end that can update as late arriving data comes in (like Materialize), and think those have there place. However for many uses of stream processing what's important is taking action once the data is complete enough, and watermarks a useful and pretty straightforward mechanism for that.
I can think of quite a few use cases (particularly in finance) where we'd want late-arrivals to be recorded and possibly incorporated into later or revised results, not silently dropped on the floor.
Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-wi...
Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still just planned for the future.
(I'm the co-creator of Arroyo, for context)
I'll try to get something set up to compare performance of the two on the same machine.
See feldera.com for more info.
Really happy to see all the activity at the new repo. For a while there I thought the project was dying...
https://www.tinybird.co/
Disclaimer: I work for Tinybird.
However, like a database we do have a serving layer (currently only in our cloud version due to its reliance on our distributed state backend: https://doc.arroyo.dev/connectors/state) so it is possible to query the results directly from Arroyo as well.
Generally you would want to use something like Arroyo when your data is too high volume to reasonably store it all in a DB like Clickhouse, or your queries are too expensive to perform on every query, as Arroyo incrementally computes the results of the query as events come in.
There's also opportunities to use these systems together: Arroyo can pre-aggregate the high volume raw data, and then it can be inserted into a Clickhouse-based system for final processing along different dimensions.
I am not sure about your premise that the operations side is difficult. It tends to be submitting a job to a cluster in Flink or Spark.
The harder barrier to entry is the functional style of transformation code. Even though other frameworks have it, I think the SQL API as the first class citizen is the bigger differentiator.
"Arroyo" is a Spanish word meaning creek, or stream
I live in a California town called Arroyo Grande ("big creek")
I wish more products would support (or at least document how to run on) Nomad.
I'm not very familiar with Benthos, but it appears to be on the simple, stateless end. Systems like that can be great if they fit your use cases—they tend to be much easier to understand and operate.
But they are not as flexible or powerful as stateful systems like Arroyo or Flink. For example, they cannot accurately compute aggregations, joins, or windows over long time ranges.
(Disclaimer: I work for Decodable, where we build a SaaS based on Flink)