HN Explorer

Show HN: We Built Vercel for Data Engineers (twitter.com)

17 points by peterhunt ↗ HN
Hi HN. Today we released Dagster Cloud to general availability [1], which includes a new feature you can try that we're calling Branch Deployments. Branch Deployments were inspired by Vercel's Preview Deployments feature and deep GitHub integration. We're hoping we can bring a similar developer experience improvement to the data domain. Let us know your feedback!

[1] https://dagster.io/blog/dagster-cloud-ga-launch

1 comment

[ 2.5 ms ] story [ 14.1 ms ] thread
bdraff
Really like their concept of assets and compute.

Defining the desired results of your pipeline, the transforms that create those assets, and then letting Dagster build/materialize those results.

I’ve spent a lot of time in other tools restarting jobs, asking questions like “did that job run?” and “is this using the latest stats output?”

Dagster’s UI of “this data pipeline is current as of 12:30” gives the answers our product team needs without pain.

And for the same reason the branch compute makes a lot of sense. “The output with the updated algorithm is Y” makes a lot more sense than having a separate experiment pipeline