Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

Thank you we appreciate the support!

Definitely a great point, adding one today

I don't think SQL can be more brittle than untyped object-oriented Python. The latter is as brittle as it gets, as we all know from ML code. SQL is common for ETL exactly because it's not brittle, (mostly) declarative and easy to test and modularize.

What Python give you is more flexibility, at the expensive of exponentially more brittleness.

This comment would be more helpful if you could summarize the pitfalls of people relying upon ELv2. My impression of these variations is that they are generally used to protect themselves from a giant corp from using it to create a cloud service of some kind?

It is open source in the sense that the source is open, you can go and look at it. It's even free open source in the sense that you can take it and use in your own, commercial project without the need to compensate its authors.

The only limit is, that the project you're building with it can't be a hosted service version of the software itself - which is, what I assume, Serra's business model will be.

I don't think that "Open Source" just means Apache 2 and MIT licensed stuff - and infact feel, that the license Serra chose is one of the most generous OSS licenses that still retain just enough rights for the authors to make a living.

I put the word "open source" in the title, and am happy to change it if someone has a better term. (I'm not up on license subtleties.)

That's a smell. I thought they basically packaged the ETL portion of DBT as open source, not the data connector implementation. I'd like it be connector agnostic so that you can choose the most suitable for your needs.

Good intentions, but perhaps wrong execution. We'll see!

Moving between Spark and Pandas can cause type casting as well. For example the range of allowable dates in Pandas is much smaller than in Spark. We completely abandoned Pandas in favor of PySpark for this reason.

It seems unnecessary to use multiple dataframe implementations when Spark is already in play.

This is a completely valid point, we'll be changing the readers to directly read into Spark. Thank for the comment!

> This license was popular w folks like apollo who arguably hijacked nearly 700 contributors efforts w a license like this. Because they are using it from start at least that won't be as bad

Which Apollo are you talking about? Only one I know is Apollo GraphQL, and their main server package seems to be MIT, so I must be looking at the wrong thing. What's the story?

It's not an open source license. Maybe that's why the quotes but let's not indulge them in their lies.

If it is really a dbt clone it is an ELT tool not ETL:

https://en.wikipedia.org/wiki/Extract,_load,_transform

https://en.wikipedia.org/wiki/Data_build_tool

It's about (big) data munging.

Sure, I’ll clarify some of the terms used in that one-liner in case it’s helpful for anyone else as well.

ETL is the process of extracting transforming and loading data from a source to a destination in a data pipeline. Spark, an engine for large scale data processing, allows us to write code that can work with large amounts of data. dbt is a tool you can use to break up your SQL scripts into smaller “models” - other SQL scripts that can be reused and tested.

We described us as an end to end because we also have extractors and loaders, whereas dbt focuses on the T ( transformation step of ETL ). Each of our steps involved in extraction, transformation and loading correspond to a specific Python object defined in our Python framework. I have also updated the README in our repo to hopefully better explain how the config file links to user defined readers, writers, and transformers.

These are all great points, and no, Disney didn't use dbt. We wanted the demo to show how you can modularize SQL into reusable objects that you can fully customize error logs for, while also adding the value of handling all steps of ETL in-house. You could definitely write this script modularly using dbt, but we feel like that value add of having connectors that easily integrate with your transforms (e2e), as well as taking the software engineering best practices that dbt applies a step further by turning each transform and connect into objects that you can test, modularize, and customize error logs for is our main differentiator.

That was my mistake, not the founders', and I've changed the term to source available (edit: now open core) to try to avoid misunderstanding. I assure you there was no attempt at fraud!

These are great links, we'll take a look

I couldn’t agree more. Except that the cost for a data tool is incredibly higher than any FE tool.

I manage both a FE team and a data team. The former spends around 1k/month on infra and hosting. While the data team easily spends 20k/month.

The gold rush is data. Build shovels.

Sure, our approach is to define Python classes to handle reusable steps for reading, transforming or loading data. For example, we have a MapTransformer, CastColumnsTransformer, GeoDistanceTransformer.

Each class specifies some configuration needed for the "step" and can then be used in the config file to construct a full ETL job. You can write unit tests for custom transformers you create as we have shown in the tests/ directory.

I have also updated the README in our repo to hopefully provide a better explanation of how our config file connects to specific Python objects.

This is super insightful thanks a ton for this gold mine.

On the python vs yaml part—definitely could've made that way more clear in the demo. Right now are framework lets you call these python objects in your yaml file, but we are working on just a python-centric implementation as well for those that do not want to interact with yamls.

On the loom and licensing choice—that's a great point. One of the main issues we ran into is getting adoption as we originally just tried licensing out the framework (mega fail ofc)—found out the hard way that no dev wants to buy something to try it out. We're definitely flexible on our license and will take all this feedback into account.

On the barrier of entry—also super insightful. We're working on a local UI offering that will be a 'config' block builder that will be free for all installs. We're implementing a DAG view similar to Airflow on the transform level. We also want to make it super easy to see your code and preview how it changes with this local UI (and have a list of all the params you need for your spark objects without having to go through the docs). We also want to flesh out more features especially on the translate side, as well as host on the cloud.

With the complexity issue that's something I ran into Disney as well! As the product grows we definitely want to flesh out our transformers based on the scripts we see. For now, the developer can make one-off transformers—we actually have a catch all "SQL transformer" for cases where you want to just pass in your sql (similar to a dbt model) and run it that way. That way it's a fail safe for if you have one specific transform that you feel is super hard to break down, you can fall back on dbt's way of just modularizing the SQL into a transform, and reference it however many times you want as an input block later on.

Thanks so much for the congrats, will definitely reach out and would love to have further discussions in the thread as will.

Yep we only showed our configuration file which instantiates the said Python objects. Definitely would've made the demo clearer, and if you want to look at the actual objects that our config is referencing, definitely check out our repo!

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

I agree. The open source flavor of dbt Core is very well-designed and almost as complete as you would want it, but any competition is good competition

Sometimes there is more to it, like pulling data from external services or running ML, but other than that, yeah it's SQL, DAGs, and cronjobs

This is a great point—in terms of dynamic tables/materialized views, the software engineering best practices ie modularization, testing, version control are not as intuitive/straight-forward to apply in comparison to dbt and Serra. We can also add a direct SnowflakeDynamicTableWriter into our framework to work with the best of both worlds!

It is likely that these will make it into dbt since there are many dbt users who also use snowflake. The main thing to consider when deciding to some something other than dbt is:

* can you apply software dev best practices, CI/CD etc * is it proprietary or can you use it with other dbs * is there a large community behind it e.g. dbt packages and dbt python libraries * will you also get docs, dq, lineage or will you need additional tools\ing * will you need to orchestrate other aspects of your data flow e.g. EL, then T, then activation, etc

Databricks also has delta live tables and for the reasons above I usually suggest people consider all of these and not just go all-in with one vendor

Greatly depends on your environment. I am thankfully in an area where there are very modest timeliness requirements. Improving the speed of a job means little to me. However, improving debugability or checkpointing when things go wrong is always valuable.

We see this working more alongside Airflow—we see Airflow mainly centered as an orchestrator/scheduler to chain together your ETL steps after you've written your data transformations and connections. With Serra, we're a flexible dev tool to write these transforms and connectors. I think you could accomplish something similar but the implementation would be unwieldy (ie breaking up 4 SQL script tasks into 40 modular transform blocks), whereas we see ourselves falling into the camp of being able to integrate with Airflow and have great error logs in those 4 SQL tasks that you have simplified into Serra configs.