Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL (github.com)
On an average, mid-scale organizations use around 40 SaaS apps. These are powerful in functionality, but limited by the quality of the product/customer data which is fed into them. The data getting synced into these tools is often incomplete, suffers from quality issues, and requires unreliable and manual imports (e.g. from CSV).
Manish and I were founding engineers at Hevodata, an ETL company, when it went from 5 customers to around 300 customers. We started seeing the trend of more and more customers wanting to move the data out of their cloud data warehouse to feed their business tools. We built a prototype to solve this for our users, but when we went deep into their use cases, we found that there were a lot of unsolved problems in this space. We also realized that activating warehouse data reliably for operational purposes was emerging as the next big trend for data-driven companies.
We did some research and came across Census/Hightouch, which were early-stage Reverse ETL cloud solutions at the time. But from our previous experience working in the ETL space, we believed that any data pipeline solution needs to be open source to cover the long list of connectors that needs to be built. So we set out to build our open source Reverse ETL solution.
With Castled, companies can create automated data pipelines to periodically sync the output of a warehouse transformation query or dbt models(on the works) to their sales, marketing, support and notification tools. We fetch only the incremental results by default on every pipeline run, which makes sure that rate limits and other constraints of the destination APIs are not breached. Our users can also set a time schedule to define the frequency of the pipeline run.
The technical challenges in building such a tool include: doing CDC (Change Data Capture) from data warehouses which do not provide a typical write ahead log; handling rate limits on destination APIs; handling deduplication of records on destination objects; failure handling and automatic retries. But the biggest challenge is the sheer number of destination app integrations that need to be supported—we are talking about tens of thousands of connectors.
Our major differentiator from Census/Hightouch is that we are open source. Our users can host Castled in their own private cloud and start operationalizing their data for free. We’ve observed that initially customers are inclined towards buying a cloud solution for their data integration needs. But once they scale up, they realize that their cloud vendor is unable to cope with the increasing number of apps getting used in the organization. They soon start building in-house data pipeline solutions or look for an open-source solution to solve their problems. Being open source, we provide the flexibility for our customers to build their own connectors rather than waiting for cloud vendors to fulfill their connector requests.
Compared with open-source alternatives (e.g. Grouparoo), we have built Castled in such a way that our community can build new connectors in a few hours. One example of this is our Castled Form Language (CFL), which helps our users auto generate extremely complex forms on the UI by writing a few Java annotations on the backend. This removes the need for a UI developer to build a new connector.
We have our Github repo here : How are you thinking of monetizing? "Being open source, we provide the flexibility for our customers to build their own connectors rather than waiting for cloud vendors to fulfill their connector requests." Why does it need to be OS? Can't a product just have a devkit that enables you to build your own connectors? The only downside is it's stuck on supporting Python2.x so you may end up wanting to properly integrate CPython eventually. Since you're targeting running Python code that doesn't exist yet and the language differences aren't huge though, I doubt most users would mind (I wouldn't). Just an idea to consider, esp for an MVP (One /upside/ is Jython is a Python2 interpreter fully written in Java, so the concurrency and performance may be better than CPython2 with its GIL) Nowadays, probably something like python then rust/go, just for community, and especially aligned on apache arrow. OSS async python / HTTP, with arrow dataplane support (fast,typed,standardized), is part of our bar for whether we consider a data proj as a core dep nowadays. A surprising amount of ETL startups are YOLO json for the dataplane, so we've intentionally stayed away due to reliability+perf heart pains. But maybe you can fake it till you make it that way too, and then hire staff to clean it up 2 years later :) Great product and very excited for this, wish I could invest in you and wish this had been around years ago when I was trying to convince Fivetran they should create reverse ETL functionality. Ps: experience is poor for https://oss-docs.castled.io on mobile, I cannot see a menu to switch pages Sorry about the docs. Haven't done much testing on smaller screens yet. I'm curious how you differentiate yourselves from Airbyte, which isn't really designed for reverse ETL but can be used for it. And do you ever see Castled supporting regular ETL? Right now there is a lot of separation in the market between ETL and reverse ETL, but it seems like a pain to maintain separate tools when you could just do both in one. I will have to check how airbyte supports both. Regarding Castled, Regular ETL is there in our mid term roadmap. https://airbyte.com/blog/airbyte-strategy-to-commoditize-all... But my personal take is that "Reverse ETL" is still a new technology in the sense that it completes the modern data stack, which is built around cloud data warehouses. Is there a way to version control the sync configurations? Any thoughts on putting that in the roadmap? I'd love to be able to put my 'Castled config' in the same repo as my dbt project, for example. I actually mean the 'definition' of the syncs themselves. I am picturing JSON or YAML that describes the source fields, their mapping to the destination fields, and any other meta about the sync: frequency, number of retries, whatever else that you could configure in the UI So when I go and update my dbt model to modify one of the tables that I am syncing from, I can make the corresponding changes to my Castled settings file, and release it all as one atomic update to my data infrastructure. It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team. But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense? Looks awesome, I am rooting for you guys! Yeah this one certainly depends on the target customer. For me, any tool that didn't have source control integration for configuration would be a non-starter. But it's quite possible that the target audience for this tool doesn't even understand the term "source control". At Grouparoo, this is a primary use case. We have a UI that engineers use locally. This helps gets things right. It outputs a JSON configuration that is checked in. When that is deployed, it does all the syncing. Do you need to know the primary key of the source table to sync? We keep snapshot tables on your data warehouse(in our own custom schema, so that you dont have to provide Castled write access to any of your production schemas). The snapshot tables are then used as the baseline to compare the query results everytime the pipeline runs. Frankly, we have not really seen a use case of transferring 5 billion rows in a Reverse ETL pipeline. This is mostly because of the fact that our destination apps are mostly transactional systems and cannot really store so much of data. For example, salesforce destination can store max 10GB of data. Because of this, we are storing the actual tuple values in the snapshot. We have easily scaled our pipelines to compute deltas from queries which returns up to 100 million records. To optimize this further, we are also considering to keep the hashes of the tuple values instead of the actual values. Yes, we need to know the primary key of your query results. This is required to handle failures and to remove the failed records from the snapshot table, so that those can be retried on the next pipeline run. It's without the tiniest sense of irony one will observe the "Signin with Google" button on the login form, too :-/ ed: although I may have ruined it for everyone, since my "team name" is now "GMAIL.COM" :-P48 comments
[ 217 ms ] story [ 1606 ms ] thread "team": {
"id": 29,
"name": "GMAIL.COM",
"tier": "Free"
}