I lead an initiative in building a data engineering program and randomly picked Airflow to orchestrate our work loads before doing a deeper dive into discovering the right solution.
We’re still running Airflow, and not because we had invested too much into it but because We found it to be the most easy to use and manageable solution.
There are a few reasons why this could be chosen over something like Airflow. They are (but not limited to): modern workflow semantics including both functional and imperative API and the ability to work with dynamically mapped tasks, a stricter / richer notion of state, data passage between tasks as a first class operation, better scheduling semantics (workflows aren’t required to have a schedule, scheduled flows can still be run off schedule, and it’s possible to have multiple runs of the same flow occurring at the exact same time), etc…
Just a tip. I tried to go into the docs to see what it actually does and couldn’t find it, just vague text about workflows. I’m not a data engineer but I am a data scientist. Tell me what kind of tools it’s going to replace front and center. Work on the elevator pitch directly on GitHub.
4 comments
[ 5.5 ms ] story [ 33.5 ms ] threadWe’re still running Airflow, and not because we had invested too much into it but because We found it to be the most easy to use and manageable solution.