I’m a huge fan of Airflow but over the course of the last few years working as a software/data engineer I’ve often needed to build pipelines that aren’t only scheduled, but event-driven.
I wrote Orkestra to scratch an itch I often have when building data pipelines.
I want the programming model Airflow provides — letting me design my workflows dynamically from code by combing operators with the >> bit shift operator — instead of doing YAML engineering.
I also want to have workflows that are triggered based on events such as those from EventBridge.
Lastly, I want to be able to scale from zero to infinity without breaking the bank.
I hope Orkestra brings you joy and addresses much of your pain when building data pipelines
1 comment
[ 4.7 ms ] story [ 14.0 ms ] threadI wrote Orkestra to scratch an itch I often have when building data pipelines.
I want the programming model Airflow provides — letting me design my workflows dynamically from code by combing operators with the >> bit shift operator — instead of doing YAML engineering.
I also want to have workflows that are triggered based on events such as those from EventBridge.
Lastly, I want to be able to scale from zero to infinity without breaking the bank.
I hope Orkestra brings you joy and addresses much of your pain when building data pipelines
https://www.knowsuchagency.com/orkestra/