We didn't look into a dataflow engine specifically, in part because we have a heterogeneous set of workfloads. Our core use case is loading mission critical data in chunks, but it is also coordinating SaaS tools and…
Interesting, Astronomer was actually my last choice for orchestrator. We went with Dagster, but I didn't want to make the takeaway "Dagster solves these problems", because it doesn't directly. Astronomer was just the…
Nope -- just MWAA, Astronomer, Dagster Cloud, and Prefect Cloud. In the past I used Argo Workflows pretty extensively and have talked about its pros and cons here: https://www.youtube.com/watch?v=-cyr_kL-9fc
Yep, that's what I tried to point out in the article.
Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We…
We didn't look into a dataflow engine specifically, in part because we have a heterogeneous set of workfloads. Our core use case is loading mission critical data in chunks, but it is also coordinating SaaS tools and…
Interesting, Astronomer was actually my last choice for orchestrator. We went with Dagster, but I didn't want to make the takeaway "Dagster solves these problems", because it doesn't directly. Astronomer was just the…
Nope -- just MWAA, Astronomer, Dagster Cloud, and Prefect Cloud. In the past I used Argo Workflows pretty extensively and have talked about its pros and cons here: https://www.youtube.com/watch?v=-cyr_kL-9fc
Yep, that's what I tried to point out in the article.
Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We…