In the streaming SQL space, I feel like at this point it's essential to start off with saying what you bring to the table compared with Materialize.
No matter what you think about how the Materialize, the company, has executed, differential dataflow should still be the standard that all streaming SQl hold themselves to.
If you can't do recursive CTE's streaming, then you are worse than Materialize and you need to show that you add something above and beyond.
Note, I don't work for, or have any affiliation with the company, I just enjoy and wish more people have read, 'Scalability! But at what COST?'.
I was hyped about materialize in the beginning but now looks like they’re just cloud focused and it’s unclear who is supposed to be their customer at all. Also I swear I saw their original docs said they can do window functions but not anymore.
Hi, author here :)
I think there are two critical differences, one on the tech side and one on the UX side-
On the UX side, we're seated behind your database, and our goal is for you to only ever interact with your original database, and never have to query us directly. This has a lot of implications when it comes to migrations on your database (you call epsio.create_view in your original DB which is transaction safe), and not having the backend interact with two databases.
On the tech side, we're built from ground zero to be above storage, whereas differential dataflow (the package Materialize is built upon) is strongly built to be above memory (we actually played with the idea of using differential with spillover to disk- was too slow). This means we actually lose when it comes to speed (e.g. we may take ~50 milliseconds more which can be a big deal in realtime applications), but have huge gains in cost, since we're trading compute for storage.
Check out our FAQ- we have a nice little table for comparison :)
I have so many issues with the underlying context of this article. Its choice of neo-christian metaphor. It’s assumptions of good vs evil. The subtle hints that they are your savior yet can’t muster more than 2 quotes on their tech from customers. The article could have easily gone with another metaphor or needle->haystack rather than invoke subliminal God complexes.
I’m also curious as to what more you bring to the table over Materialize. CTE’s, etc. How about a blog post where you leave the holier-than-thou christian symbolism for actual tech features and why one should consider you for their data engine needs. Beyond trying to see if The Lord has admin access.
I don’t find cults humorous. I know what you are saying though I just think there’s dozens of other ways that could have been said. It shows a lack of professionalism in their communications department to me. Be that as it may, I still read the article, I browsed the site, I looked for more tech evidence and reasoning for adoption and found it lacking.
There's no reference to or invocation of cults, just a playful use of the cartoonish idea of evil people. I'd find it distasteful if there were like, references to kool-aid or Manson or whatever, but there aren't. Is it your position no one should poke fun at the idea of "evil"? Honestly that just seems like no fun.
Belief in a mystical Lord is a cult belief. Yes they use it in satire but I don’t think religious tongue in cheek has any place in the business world unless your business is selling snake oil or bigotry.
There's a pretty big difference between what most people mean when they say "cult" (compounds, suicide pacts, etc) and your average person who goes to church/mass/temple once a week.
Like, feel free to be a big stick in the mud about this, but this is a lighthearted example. It definitely doesn't mean their product is trash or their business is fraudulent. That's bonkers.
Where Religion is communicated as as a proxy for cult, it seems to be based on an experience and/or assumption that all religions are the same because an individuals person experience of one religion obviously represents the totality of all religions and the only experiences that are possible.
It’s accidental righteousness to carry on this evangelical view even when dissociating from one particular experience or thread
What I’m more allergic to is people bashing other over the head with their personal interpretations.
So in a way my allergy is to righteousness.
Say you or I believe something is a cult. It might even be true. But still it’s one’s personal interpretation on what a cult is, and if something is a cult and furthermore you may be trying trying to externally validate your world viewpoint about religion at the expense of others, which can be another form of evangelism, and might be a remnant of said cult.
There’s lots of ways to bring up about cults. Righteousness ain’t it.
To help anyone read in between the lines, there’s no need to let religion trigger you.
If religion is a dominant world viewpoint on god, to some Atheism is a god centric view too (god doesn’t existing) is too. Then people will wiggle into spirituality and agnosticism, and that too can be seen as being subjective and a deeply personal interpretation.
So most of the world could probably agree that no belief owns the exclusive path to kindness or goodness. At the same time if something is working for someone and something else works for you it seems plausible something else might work for someone else. Righteousness is an issue tho.
This is entirely the kind of discussion that they should avoid by keeping the blog about the tech. The choice to use religious language and topics just invites all of this and takes away from the tech.
I see religion or “mass think” as a cult. No matter its source, doctrine, humble beginnings, or commercial perversion. Free thought, respect for all life, and a moral compass on the “Lawful Good” side. I can take metaphor when the content suggests it or if it adds a layer to the story. In this context, it’s completely out of place.
I also respect your right to follow the group think. If you find comfort in turning off your brain, go for it. People do it in different ways. For me, riding a wave or skateboarding has a way of turning off my brain and rejuvenating my soul. I studied Catholicism, Buddhism, Judaism, Tao, I Ching, and many more. Today, in the name of religion, there’s a war going on: so to use this kind of rhetoric or metaphor for explaining your blazingly fast sql engine is just extremely poor taste. Yes, I will bash religions, all of them, until they stop waging war in the name of them.
That’s ok that you see it that way.
It’s not a metaphor I would use either.
Saying you respect someone’s right to believe and then not carrying yourself respectfully towards people who don’t think like you, is kind of funny.
The continued reaction to it by calling it a cult is also a bit extreme, and I’m not sure if you grew up around cult like belief.
Riding that inner skateboard as you said is what life is about.m and whatever does that for someone without harming others, they should do.
It seems like demonstrating a practice of respect is something that is eluding your expression at the moment. When we contemplate and reflect on what respect is, we can explore if it really is dismissing others who don’t agree with the way we see things by calling others ‘cult’ like, to avoid the risk of coming across as a bit deluded.
HN by your definition is cult like and group thinking.
If you come after some beliefs of others, indirectly dismissing them, its not respect, it’s actually the opposite, it’s feigned tolerance that easily can give rise to other forms of othering, bias towards others and a lot worse as we are seeing in the world.
Most people are in pursuit of a personal truth, and a dominant world viewpoint that enables that. What it consists of can vary.
Where you call others out to externally validate your own beliefs, it isn’t respectful.
Bad activities of any group should be called out on the basis of them being bad.
Things that are someone else’s personal practice that is no harm to you is up to them. This remains about being triggered and not wanting to see something a certain way.
Group think is a human feature and is something everyone is a part of including you and I.
Not all religions wage war, your anger is kind of misplaced. Respectfully, you haven’t studied as far and wise as you may think you have.
Many tyrants have historically hijacked many things for political purposes and gain, including religion. History is wide open to soak up that knowledge.
A “war” going on today isn’t religious, it’s about political power. And it’s awful that people can abstract away the humanity of others.
I do hope you see a small version of that is abstracting someone and their beliefs away is kind of dehumanizing them. If I call you hopelessly lost in a cult, it’s easier to dissociate from you as a human being, and justify disdain, mistreatment and much more.
They have a table comparing what they are doing vs materialize in the FAQs in the docs (not super easy to find) - https://docs.epsio.io/faq/
On a quick read, seems like the core difference is around whether you query the streaming engine (Materialize) or still interact with your db. Other differences are then downstream from this.
I found the references to god and religion in a technical article off-putting as well. I tried hard to think of it as an attempt at humor, but I could not shake the feeling that the writer was a very religious person. I tend to value the
technical opinions of deeply religious people less.
If you actually believe in magic, then I am going to have a hard time trusting you with anything critical.
Isn't this what is done by some SQL engines with materialized views?
They update view data (counts and group by and sums) as the source table data changes. But not by rerunning the view, but smartly updating its contents based on the change that happened.
There are tons of limits in these views, as they have to provide stable results.
The fact that multiple transactions can see different
states of the data means that there can be no
straightforward way for "COUNT(*)" to summarize data across
the whole table. PostgreSQL must walk through all rows to
determine visibility.
> By default, materialized views in PostgreSQL are not automatically refreshed. If the underlying data changes, the materialized view remains stale until it's refreshed.
Can’t this be somewhat alleviated by running a job native on Postgres or some external one to refresh the views on a schedule? It’s not ideal or realtime but better than having it done as-hoc.
Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.
Hi, author here :)
In general, many databases have attempted something similar- but because "batch query" engines are so monumentally different than incremental engines, it becomes a very tough task and ends up being a hill of patches- which leads to many limitations.
The easiest example I can think of is the direction of the flow of data. Postgres, for example, works from "top to bottom", where you start with the result node which runs the node underneath it and so on. Each node continues requesting information from the node underneath it until it returns Null, meaning it finished. This works very well with a snapshot of the data- but if you want to maintain a high throughput with a constant stream of changes, you're going to want to work in the opposite direction- this would mean either overhauling the very way Postgres works, or adding a completely different engine into Postgres.
One big one I've seen are online ML models. You essentially have a feature store built on SQL ETLs. The ML engineers utilize this to build models, and deploy those to production.
However when using the models for prediction, the data hasn't been ETLed (think: credit modeling). One solution here is to use the same SQL for the ETL and for the online transformation to query the model. It can guarantee the biz logic in the features matches between training and online.
A large variety of use cases really, e.g. fraud detection via pattern matching on a stream of credit card transactions, the creation of denormalized views (seeing this a lot in the context of Debezium change event streams from RDBMS), updating real-time dashboards e.g. based on incoming order data, etc. I don't think it depends at the size of the company, we see users of all kinds of sizes.
(Disclaimer: I work for decodable.co, a managed stream processing platform leveraging Flink SQL)
On the topic of streaming SQL, I found this new startup because one of their team was making a ton of contributions to "sqlparser-rs" and Arrow Datafusion.
I thought it looked interesting (even though I personally don't have a use for it) and the technology it's built on is solid:
- Materialize
- Flink SQL
- Arroyo
- Readyset
- RisingWave
- Timeplus
- Pathway
- Dozer
- ReadySet
- Snowflake dynamic tables
- Native materialized views in OLTP databases
- Just having a stack of views in your db
- Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart
I'm with Arroyo [0] — thanks for the mention! I'd be interested to see someone from Epsio chime in with where exactly they're positioning, but you're right that this is (recently) a very crowded space.
I think you can somewhat arbitrarily draw a line between systems like Materialize/RisingWave that are focused on materialized view maintenance (often reading change feeds from your OLTP database) and stream processors like Flink/Arroyo that are focused on supporting more operational use cases and work with a lager variety of sources and sinks.
Epsio seems like it's working primarily in the former mold, with fast incremental computation of materialized views. Unlike Materialize/RisingWave it seems to be designed to run in front of your database, with all queries going through it.
ReadySet is a really cool project in the query caching/materialized view space that I think doesn't get enough attention. Rather than making you define your materialized views ahead of time, it acts as an automated caching layer on top of postgres/mysql that performs incremental computation for components of query graphs.
As someone who's been in the streaming space for years, it's really exciting to see so much energy in the space in the past couple of years after a long period of stagnation, with everyone trying to figure out the programming and deployment models that make the most sense.
Most folks right now are gravitating towards materialized views as the model, largely because it's easy and familar for users. But ultimately I think this approach will end up too limited for most use cases and will remain valuable but somewhat niche.
It's based off of the DBSP paper, which counts Materialize's Frank McSherry as an author. Definitely not ready for the prime time, but any feeback welcome!
I didn't - I wanted to build something I could understand, and the rust implementation comes with a load of (reasonable) complexity, some resulting from the execution model, some from performance requirements.
stepping has 3 Zset implementations - in memory, sqlite and postgres. Currently considering whether to write a small rust one.
I am often confused on how things like imperative stream processing frameworks are supposed to work with the streaming database, if at all. Like what are you supposed to interact with the streaming database with? Is it only for streaming ETL? Do you need another application layer that can natively integrate, orchestrate and function. Maybe using a stream processing framework with a streaming database could facilitate a full application stack.
I didn’t care for all the religious stuff in the blog post and there’s no run-it-myself container or method that I can tell. I have to fill out a form to “get started” which flow is very much “we don’t have a pricing page we have a contact us for more info” corporate selling vibe. I don’t care for it. So I’ll not invest time in playing with it and maybe eventually selling it internally to my team.
69 comments
[ 3.6 ms ] story [ 812 ms ] threadNo matter what you think about how the Materialize, the company, has executed, differential dataflow should still be the standard that all streaming SQl hold themselves to.
If you can't do recursive CTE's streaming, then you are worse than Materialize and you need to show that you add something above and beyond.
Note, I don't work for, or have any affiliation with the company, I just enjoy and wish more people have read, 'Scalability! But at what COST?'.
Are you sure they don’t?
> It seamlessly integrates with your existing database, saving you from complicated setup procedures.
So maybe their advantage over Materialize is that this can work on top of whatever database you already have?
I hope more companies can be built around timely-dataflow.
On the UX side, we're seated behind your database, and our goal is for you to only ever interact with your original database, and never have to query us directly. This has a lot of implications when it comes to migrations on your database (you call epsio.create_view in your original DB which is transaction safe), and not having the backend interact with two databases.
On the tech side, we're built from ground zero to be above storage, whereas differential dataflow (the package Materialize is built upon) is strongly built to be above memory (we actually played with the idea of using differential with spillover to disk- was too slow). This means we actually lose when it comes to speed (e.g. we may take ~50 milliseconds more which can be a big deal in realtime applications), but have huge gains in cost, since we're trading compute for storage.
Check out our FAQ- we have a nice little table for comparison :)
Do you support Common Table Expressions?
What kind of overhead have you seen?
Does this negate the need for indexes in some cases?
I’m also curious as to what more you bring to the table over Materialize. CTE’s, etc. How about a blog post where you leave the holier-than-thou christian symbolism for actual tech features and why one should consider you for their data engine needs. Beyond trying to see if The Lord has admin access.
The world is not responsible to not trigger us, as I’m sure references to some things do.
Like, feel free to be a big stick in the mud about this, but this is a lighthearted example. It definitely doesn't mean their product is trash or their business is fraudulent. That's bonkers.
It’s accidental righteousness to carry on this evangelical view even when dissociating from one particular experience or thread
But, let’s do yoga.
What I’m more allergic to is people bashing other over the head with their personal interpretations.
So in a way my allergy is to righteousness.
Say you or I believe something is a cult. It might even be true. But still it’s one’s personal interpretation on what a cult is, and if something is a cult and furthermore you may be trying trying to externally validate your world viewpoint about religion at the expense of others, which can be another form of evangelism, and might be a remnant of said cult.
There’s lots of ways to bring up about cults. Righteousness ain’t it.
To help anyone read in between the lines, there’s no need to let religion trigger you.
If religion is a dominant world viewpoint on god, to some Atheism is a god centric view too (god doesn’t existing) is too. Then people will wiggle into spirituality and agnosticism, and that too can be seen as being subjective and a deeply personal interpretation.
So most of the world could probably agree that no belief owns the exclusive path to kindness or goodness. At the same time if something is working for someone and something else works for you it seems plausible something else might work for someone else. Righteousness is an issue tho.
I Generally people can’t openly entertain a viewpoint that isn’t theirs as much as they’d like.
This was started by someone else so you are welcome to speak to them.
All I see is software developers wanting to create entire working worlds and universes :)
I also respect your right to follow the group think. If you find comfort in turning off your brain, go for it. People do it in different ways. For me, riding a wave or skateboarding has a way of turning off my brain and rejuvenating my soul. I studied Catholicism, Buddhism, Judaism, Tao, I Ching, and many more. Today, in the name of religion, there’s a war going on: so to use this kind of rhetoric or metaphor for explaining your blazingly fast sql engine is just extremely poor taste. Yes, I will bash religions, all of them, until they stop waging war in the name of them.
Saying you respect someone’s right to believe and then not carrying yourself respectfully towards people who don’t think like you, is kind of funny.
The continued reaction to it by calling it a cult is also a bit extreme, and I’m not sure if you grew up around cult like belief.
Riding that inner skateboard as you said is what life is about.m and whatever does that for someone without harming others, they should do.
It seems like demonstrating a practice of respect is something that is eluding your expression at the moment. When we contemplate and reflect on what respect is, we can explore if it really is dismissing others who don’t agree with the way we see things by calling others ‘cult’ like, to avoid the risk of coming across as a bit deluded.
HN by your definition is cult like and group thinking.
If you come after some beliefs of others, indirectly dismissing them, its not respect, it’s actually the opposite, it’s feigned tolerance that easily can give rise to other forms of othering, bias towards others and a lot worse as we are seeing in the world.
Most people are in pursuit of a personal truth, and a dominant world viewpoint that enables that. What it consists of can vary.
Where you call others out to externally validate your own beliefs, it isn’t respectful.
Bad activities of any group should be called out on the basis of them being bad.
Things that are someone else’s personal practice that is no harm to you is up to them. This remains about being triggered and not wanting to see something a certain way.
Group think is a human feature and is something everyone is a part of including you and I.
Not all religions wage war, your anger is kind of misplaced. Respectfully, you haven’t studied as far and wise as you may think you have.
Many tyrants have historically hijacked many things for political purposes and gain, including religion. History is wide open to soak up that knowledge.
A “war” going on today isn’t religious, it’s about political power. And it’s awful that people can abstract away the humanity of others.
I do hope you see a small version of that is abstracting someone and their beliefs away is kind of dehumanizing them. If I call you hopelessly lost in a cult, it’s easier to dissociate from you as a human being, and justify disdain, mistreatment and much more.
Being divisive is not good for society.
Everything you write on a company blog is a representation of the company's identity. The company is implicitly saying "I support this".
On a quick read, seems like the core difference is around whether you query the streaming engine (Materialize) or still interact with your db. Other differences are then downstream from this.
If you actually believe in magic, then I am going to have a hard time trusting you with anything critical.
They update view data (counts and group by and sums) as the source table data changes. But not by rerunning the view, but smartly updating its contents based on the change that happened.
There are tons of limits in these views, as they have to provide stable results.
See ms sql documentation on limits of streamed/indexes sql views https://learn.microsoft.com/en-us/sql/relational-databases/v...
Also worth noting Delta Live Tables, another very popular streaming SQL implementation. https://www.databricks.com/product/delta-live-tables
Seems the author made a lot of assumptions without a lot of research.
For one example, I would strongly doubt that count(*) without a where clause isn’t already highly optimized.
It's actually not well optimized in PostgreSQL because of MVCC. See https://wiki.postgresql.org/wiki/Slow_Counting
TL;DR:
If my RBDMS adds materialized views, I want it to help me with it lol
https://github.com/timelydataflow/differential-dataflow
https://materialize.com/
The easiest example I can think of is the direction of the flow of data. Postgres, for example, works from "top to bottom", where you start with the result node which runs the node underneath it and so on. Each node continues requesting information from the node underneath it until it returns Null, meaning it finished. This works very well with a snapshot of the data- but if you want to maintain a high throughput with a constant stream of changes, you're going to want to work in the opposite direction- this would mean either overhauling the very way Postgres works, or adding a completely different engine into Postgres.
However when using the models for prediction, the data hasn't been ETLed (think: credit modeling). One solution here is to use the same SQL for the ETL and for the online transformation to query the model. It can guarantee the biz logic in the features matches between training and online.
(Disclaimer: I work for decodable.co, a managed stream processing platform leveraging Flink SQL)
Imagine you have 10K/s of insert/deletes on that table with replication.
not clear if it is that much
I thought it looked interesting (even though I personally don't have a use for it) and the technology it's built on is solid:
https://www.synnada.ai/
- Materialize - Flink SQL - Arroyo - Readyset - RisingWave - Timeplus - Pathway - Dozer - ReadySet - Snowflake dynamic tables - Native materialized views in OLTP databases - Just having a stack of views in your db - Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart
I think you can somewhat arbitrarily draw a line between systems like Materialize/RisingWave that are focused on materialized view maintenance (often reading change feeds from your OLTP database) and stream processors like Flink/Arroyo that are focused on supporting more operational use cases and work with a lager variety of sources and sinks.
Epsio seems like it's working primarily in the former mold, with fast incremental computation of materialized views. Unlike Materialize/RisingWave it seems to be designed to run in front of your database, with all queries going through it.
ReadySet is a really cool project in the query caching/materialized view space that I think doesn't get enough attention. Rather than making you define your materialized views ahead of time, it acts as an automated caching layer on top of postgres/mysql that performs incremental computation for components of query graphs.
As someone who's been in the streaming space for years, it's really exciting to see so much energy in the space in the past couple of years after a long period of stagnation, with everyone trying to figure out the programming and deployment models that make the most sense.
Most folks right now are gravitating towards materialized views as the model, largely because it's easy and familar for users. But ultimately I think this approach will end up too limited for most use cases and will remain valuable but somewhat niche.
[0] https://github.com/ArroyoSystems/arroyo
We Built a Streaming SQL Engine (epsio.io) 83 points by croes 10 hours ago | flag | past | 29 comments
How We Built a Streaming SQL Engine (epsio.io) 8 points by nightmaremard 3 days ago | flag | past | discuss
How To Build a Streaming Engine (epsio.io) 4 points by daymareman 20 days ago | past
How We Built a Streaming SQL Engine (epsio.io) 1 point by dkgs1998 23 days ago | past
How we built a Streaming SQL Engine (epsio.io) 2 points by giladkkl 37 days ago | past
How We Built a Streaming SQL Engine (epsio.io) 2 points by dkgs1998 38 days ago | past
https://stepping.site
It's based off of the DBSP paper, which counts Materialize's Frank McSherry as an author. Definitely not ready for the prime time, but any feeback welcome!
stepping has 3 Zset implementations - in memory, sqlite and postgres. Currently considering whether to write a small rust one.
There's some more details here: https://stepping.site/docs/internals/how-it-works/
I have a couple of dbsp questions if you're feeling generous and have 10mins free - reach out ojhrussell at gmail