Airflow 2.0 is a major release that's six months in the making. In addition to being much, much faster, this release includes HA compatibility, a new REST API, a refreshed UI, and a much simpler DAG creation experience.
We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice.
We'd really appreciate if the ECS Operator could be given a bit of attention:
Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish.
---
Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!
Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community.
One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.
Ditto, running Airflow on AWS ECS Fargate serverless. We did this prior to AWS announcing their Managed Workflows for Apache Airflow[1]. Do you know if and when AWS will be making Airflow 2.0 available in their managed service?
Have you tried Managed Workflows for Apache Airflow?
I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.
Are there any plans for DAGs packaging in docker containers similar to what Prefect does?
Would be perfect to have separate dependencies for different DAGs, otherwise we always end up with a pile of everything ever needed with no clear way to remove obsolete packages from setup.
So there are a few options for that if you're interested!
1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or if you want to handle baking in DAGs for different images).
2. You can use custom images in KEDA queues. This way you can simply point to a queue for all tasks in that DAG and they will run in that environment.
3. You can use the k8spodoperator. Now that the k8spodoperator allows for templating, it would be pretty easy to create a template for a pod and just inject different commands for different steps.
That's not exactly what I was looking for, though. Because every listed approach injects technical complexity in the middle of my business logic.
I.e. if I have two consequent tasks I have to define them as a separate scripts or commands, package them, upload and then orchestrate them in a completely different place.
While in Prefect I have all the niceness of writing almost plain Python (as with new tasks API in Airflow), then I can package and distribute the whole thing in docker image with a single command. It really matters!
Ohh you are welcome, join the slack channel and ask for help. The community is growing everyday - here are some examples of using it in python https://flytecookbook.readthedocs.io/en/latest/
I've been using Airflow for the past 6 months extensively in my projects and it is such a pleasure to use. I am excited to try out this release. Had a little trouble integrating Selenium and webDriver using BashOperators that run individual Python scripts and should probably switch to PythonOperators soon.
If you're switching to python operators you should check out the TaskFlow API. You can basically build python operators with just python functions and decorators.
Can you still chain dependent tasks together with the cute ">>" syntax? I like that, makes reasoning about task orderings and co-dependent tasks pretty easy to write and think about.
Thank you for clarifying, that scared the heck out of me.
Is there an official guide of how to update from 1.10.x to 2.0, or do we still need to do the 1.14 update, then move over? I'm interested in updating ASAP, but scared to break my production.
Congratulations! I came across Airflow being used mostly for ETL space. But are there any other use cases that it is suitable for? Is it comparable to (or build) a BPMN style workflows? Thank you.
I would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?
If you can turn it into a bunch of python (or bash) scripts that should be scheduled after each other you could probably use airflow. I'm not saying you should, but it's perfectly doable. Airflow works best when you're free to restart jobs though, that's probably something to keep in mind.
And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).
I looked at Airflow a long time ago, and it seemed focus on running tasks based on a time-based schedule. For example, if I needed to create a daily sales report every night, Airflow would be great because I could set up a timer to kick off a reporting process every night. Then I could do things like rerun the last 5 days of reports if needed, etc.
I was more interested in event-based tasks though. For example, a sale happens, and then a report about that individual sale gets generated. At the time, this seemed to require some Airflow hacks and would cause me to miss out on some of its features. Is this still the case?
A sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.
When I looked at it, it looked like sensors were all polling based. Things like wait until another task completes or this file appears. Is there a way to make them push/event based?
We didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I believe this is on the docket for 2.1 or 3.0 depending on whether it requires a breaking change (that said we plan to release much more frequently going forward so we're planning to have this feature in 2021)
I recommend checking out Conducto. It handles dependencies well by leveraging Docker and it is very easy to get started. It can also handle super complicated pipelines that required scaled up processing by just changing one input on the command line.
I wonder if there's anything different about the Airflow community as opposed to other Apache communities with regards to UI design management.
ASF communities work hard to be open to contributions from any and all parties, which is important for avoiding alienation of minority factions and keeping the community from fragmenting, and also for maintaining project independence and not getting captured by a single dominant entity. As a side effect, contributions are often negotiated but rarely rejected, which exerts pressure in the direction of feature bloat: the acute needs of the contributor tend to outweigh the diffuse benefit of simplicity.
Such pressure is not impossible to resist, but it usually takes dedication and and superior negotiation skills from core maintainers. A modular architecture is also very helpful for accommodating contributors without compromising usability, which is why so many successful ASF projects use plugins.
could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )
On the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will help out with a demo if you want help on that)
I'd not heard of airflow before. Some of the other comments mention using it for ETL jobs. The webpage makes it sound like a generic job scheduler.
Jenkins is also a generic job scheduler, but it is primarily used for CI/CD. It seems like there's some overlap here though; I'm interested to hear other peoples thoughts on "I have a bunch of jobs that need to be run; they aren't necessarily ETL or CI/CD; what tool is good for this, and how would I differentiate between them."
It seems that mostly people use whatever they have setup, so if someone already does CI/CD through jenkins, they will add other unrelated jobs to jenkins just to avoid setting something else up. Any thoughts on if things are being left on the table here, as even generic tools can make you feel like you are "swimming up stream" if you are doing things that the rest of the community is not with them.
Here is my personal opinion of some Pros and Cons for the types of jobs that are in the fuzzy area of overlapping functionality.
Jenkins gui and interface handles lots of standalone jobs more nicely.
Jenkins has much better support for parameterised jobs that can be kicked off manually.
Airflow can handle dependencies between jobs in a much better way. Nicely defining and visualising dags of job really is the killer feature.
Browsing task/job logs is nicer in Airflow IMO.
Airflow scheduler is flaky - hopefully better in 2.0.
Airflow has much more “magic” than Jenkins this is often infuriating.
All that said, my preference is to move jobs to Airflow. Getting a nice gui for manually triggered jobs in airflow is/was the only large missing piece for me.
I love airflow because it shows you the tasks in a tree view, and dag view, and you can trigger jobs manually.
The UI is fine, but looks to be improved in 2.0.
Main improvement I am looking forward to are the scheduler changes, I run a few jobs that trigger every minute in airflow, and it's definitely not intended for that.
Looks like a lot of options that were introduced in Prefect are now being introduced in Airflow. Obviously Prefect borrows heavily from Airflow's lineage. Seems like a little competition has been good for the space.
Here here! The docs don't really make it clear what Airflow is for someone who hasn't already used or been exposed to it. Is it a job scheduler like Rundeck? Something else?
It's a job scheduler that runs directed acyclic graphs. A key feature of Airflow is you can more or less write your workflow naturally in python, and it will create the workflow graph for you. It includes a web interface for managing your workflows, a scheduler, and much more.
81 comments
[ 1.4 ms ] story [ 134 ms ] threadhttps://user-images.githubusercontent.com/3267/96045809-852d...
Feel free to AMA about Airflow's new features/the roadmap going forward!
We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice.
We'd really appreciate if the ECS Operator could be given a bit of attention:
Running a task on FARGATE_SPOT containers is a cheap, convenient option, but it requires passing capacityProviderStrategy in. https://issues.apache.org/jira/browse/AIRFLOW-6604
Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish.
---
Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!
One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.
[1] https://aws.amazon.com/managed-workflows-for-apache-airflow/
I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.
Would be perfect to have separate dependencies for different DAGs, otherwise we always end up with a pile of everything ever needed with no clear way to remove obsolete packages from setup.
1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or if you want to handle baking in DAGs for different images).
2. You can use custom images in KEDA queues. This way you can simply point to a queue for all tasks in that DAG and they will run in that environment.
3. You can use the k8spodoperator. Now that the k8spodoperator allows for templating, it would be pretty easy to create a template for a pod and just inject different commands for different steps.
Hope that helps!
That's not exactly what I was looking for, though. Because every listed approach injects technical complexity in the middle of my business logic.
I.e. if I have two consequent tasks I have to define them as a separate scripts or commands, package them, upload and then orchestrate them in a completely different place.
While in Prefect I have all the niceness of writing almost plain Python (as with new tasks API in Airflow), then I can package and distribute the whole thing in docker image with a single command. It really matters!
I seriously love that you're an Airflow user since your spark talks first got me into OSS.
Edit:
Sorry I misspoke here
The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.
is totally fine Is no longer allowed. Apologies for the miscommunication.The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.
is totally fine Is no longer allowed. Most of your DAG should be completely fine. Apologies for the miscommunication.Is there an official guide of how to update from 1.10.x to 2.0, or do we still need to do the 1.14 update, then move over? I'm interested in updating ASAP, but scared to break my production.
https://stackoverflow.com/q/65345789/220997
THIS! Mainly looking forward to the scheduler HA improvements! That has been the biggest pain in a self managed Airflow environment.
And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).
If you're comfortable with docker-compose you can probably find an example setup and get it running in a few minutes though.
I was more interested in event-based tasks though. For example, a sale happens, and then a report about that individual sale gets generated. At the time, this seemed to require some Airflow hacks and would cause me to miss out on some of its features. Is this still the case?
Of course a lot of people think of Airflow as 'simpler' than Luigi, because Airflow is all graphical. YMMV.
https://aws.amazon.com/managed-workflows-for-apache-airflow/
ASF communities work hard to be open to contributions from any and all parties, which is important for avoiding alienation of minority factions and keeping the community from fragmenting, and also for maintaining project independence and not getting captured by a single dominant entity. As a side effect, contributions are often negotiated but rarely rejected, which exerts pressure in the direction of feature bloat: the acute needs of the contributor tend to outweigh the diffuse benefit of simplicity.
Such pressure is not impossible to resist, but it usually takes dedication and and superior negotiation skills from core maintainers. A modular architecture is also very helpful for accommodating contributors without compromising usability, which is why so many successful ASF projects use plugins.
Airflow is a backend project built by backend engineers.
Most UI people don't use Airflow or know what it is.
@ryanhamilton is the first front-end dev to become a committer on the project and that JUST happened a few months ago.
If I may hijack this thread for a small feature request plea. I wish that the gantt chart is exposed as an API.
Or, simply support alerting directly inside the gantt chart.
It would make monitoring and alerting much much simpler.
could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )
Jenkins is also a generic job scheduler, but it is primarily used for CI/CD. It seems like there's some overlap here though; I'm interested to hear other peoples thoughts on "I have a bunch of jobs that need to be run; they aren't necessarily ETL or CI/CD; what tool is good for this, and how would I differentiate between them."
It seems that mostly people use whatever they have setup, so if someone already does CI/CD through jenkins, they will add other unrelated jobs to jenkins just to avoid setting something else up. Any thoughts on if things are being left on the table here, as even generic tools can make you feel like you are "swimming up stream" if you are doing things that the rest of the community is not with them.
Here is my personal opinion of some Pros and Cons for the types of jobs that are in the fuzzy area of overlapping functionality.
Jenkins gui and interface handles lots of standalone jobs more nicely.
Jenkins has much better support for parameterised jobs that can be kicked off manually.
Airflow can handle dependencies between jobs in a much better way. Nicely defining and visualising dags of job really is the killer feature.
Browsing task/job logs is nicer in Airflow IMO.
Airflow scheduler is flaky - hopefully better in 2.0.
Airflow has much more “magic” than Jenkins this is often infuriating.
All that said, my preference is to move jobs to Airflow. Getting a nice gui for manually triggered jobs in airflow is/was the only large missing piece for me.
So there is a GUI, but it isn't nice. Am I reading that correctly?
Or perhaps the functionality doesn't exist at all? Or can be done at the command line or API?
The UI is fine, but looks to be improved in 2.0.
Main improvement I am looking forward to are the scheduler changes, I run a few jobs that trigger every minute in airflow, and it's definitely not intended for that.