There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version
If you think of the major cloud providers as "operating systems" (and you should), this is no different than a piece of software having a Mac version, a Windows version, and a Linux version.
Or people working to make some *nix tools work on different distributions.
Only so long as you are able to easily interop one tool from one cloud with the others without major consequences. If that was true, then you can freely mix and match cloud services from different vendors.
Alas, this relies on many things, not the least of which are peering agreements between cloud vendors that do not punish the consumer for using a service on one cloud with services from another.
It is possible to an extent today, but for competitive reasons, cloud companies do not seem to have a built in incentive to collaborate. Perhaps the cost of switching and lock-in will force cloud providers to work together in the long one, else they risk alienating customers, and then lack the ability to capture a larger market.
In other words, competition is good as long as people can realistically take advantage of it. I think cloud vendors have an obligation to see that this competition is encouraged and supported.
What is your ideal solution in these cases? In my experience, large enterprise customers (among others) tend to prefer managed versions of software where available. Cloud agnostic/interop versions would certainly seem like a good idea but I wonder what effect that would have on pricing and growth of the product.
It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge it, and use it - to the exclusion of all other implementations - not because of bias, but because of the truth underlying our meritocracy.
The question for me is - when we engage in said meritocracy, with all these BDFLs calling the shots, will they be the ones to negotiate with Jeff, Bill, and Larry? Or will we?
I haven't tried it yet but this honestly looks pretty close to vanilla, the post even says that they contributed their patches to upstream. What differences have you spotted?
Of course their storage backend is going to be S3 and they are going to send logs to Cloudwatch, we have been doing it in a similar way for quite some time and it is what I expect from a solution managed by AWS.
I respectfully disagree. Not only do cloud native versions of an OSS encourages innovation and enrichment, it also lowers the operational cost of running it in-house and aligns closely to Devops principles.
so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well.
Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably.
It'd be really helpful if there were an article that folks could read.
Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video.
This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
Because people use it. It's been around a while and has an ecosystem around it. (This is the interesting thing about AWS, they keep the focus on what people do use, rather than some opinionated idea of what they should use.)
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with.
I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.
Lmao this is a great comment - I have found so many bizarre eccentricities with Airflow, but it is still, in general, quite nice to build with. What are you thinking about?
I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x).
I realize you are buying not just the compute, but the management, but that ends up being something in the cost range of $300-$500 or so per month for the airflow management part of it. Seems a bit steep. $50-$100/mo would be a no brainer for us. For some orgs I can see this being a great solution, but its not really friendly for the little guy (with a min price of $350/mo).
Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)?
I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ against those outages.
I used to manage an airflow deployment (of which my team was the primary consumer), and it was not enjoyable in the least.
For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely.
But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.
(am OP) We haven't had a major outage in 3 years of using airflow. Some issues, and it consumes some time, but so would managed airflow in all likelihood. Most issues are related to "airflow configuration type things" that presumably this solution would not fix.
I am really just surprised it is not a "no brainier price". It's a tough sell. I am sure it's valuable for some people, I just can't really justify using it. I think value to me is like $100/mo (as random airflow user on the internet).
not OP, nor do we use airflow (not sure there is a fit for us) - but $300-500 for something that could cost our modest company an equivalent half day of human time to mitigate, with possible tens of thousands in lost revenue if it occurs at the worst possible time, seem like a win-win peace of mind proposition.
Sure. The right circumstances this can make sense. But at a high level of your SAAS starts at $500/mo, you are selling to mid market and enterprises, not startups, almost universally. I was hoping for a product that would make sense for smaller companies. The list of business critical software we pay less for is long (GitHub, credit card processor, email, sms, managed redis, managed postgres, etc).
They can certainly sell this for what they want, I just won't buy at this price.
i guess i really don't understand what separates a startup from enterprise...i know my particular company is nowhere near enterprise level...at least as far as i know of it, but i also know we're not a startup.
edit: in this sense...i know the fundamental differences otherwise
and i totally get where you're coming from - i'm the point guy on validating any new integrations for what we do. i was just commenting on the cost/benefit of having someone fix a home rolled version vs their managed and if it'd save money
I really wish managed airflow instances were cheaper for smaller companies. I built my own using spot instances and it's so affordable compared to astronomer and the others.
So far it's been very low maintenance - outside of the few random scares where dag logs filled up my server - but then I researched and found maintenance Dags that prevent a lot of issues.
Wondering when my next outage will be is always fun, but it's been pretty stable so far.
E: I know and appreciate the tech they put into it. It's just too high of a price for me once I get the workers added. I still want migrate mine to fargate workers at some point though.
Nobody is getting fired (at our org at least). We have to purchase support for Airflow due to security policy. AWS is much cheaper than any alternatives.
Our security posture requires that I purchase support for all software that we run in production. There are very few organizations offering support for Airflow and they aren't cheap.
Want to shout out an alternative Python open source workflow orchestration tool, Prefect https://www.prefect.io/
It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.
Most of the time this doesn’t have to be different, the conversation is likely a big enterprise customer says “do you have managed airflow? If not we might consider move(a little/some/all) of our workload to Google Cloud b/c they have it”
AWS: “we will build one in 3 months”
See. nobody care about the differentiator in big 2B enterprise business . It’s more about trust and migration cost
Confluent seems to be thriving ever since AWS launched their own Kafka managed service. With that as an example, this could be a good thing for Astronomer.io.
There's no mention of AWS's own existing workflow management systems - Step Functions and AWS Glue. Would be interested to see AWS's own advice on when to use one over the other.
This was developed internally by the Orchestration organization - that builds Step Functions and maintains AWS Simple Workflow [1], or have you forgotten the original AWS workflow system :)
I don't think of Glue as a generic workflow system the way the others are - it's definitely much more optimized for ETL use cases.
With time, I'm sure there'll be more detailed guidance on Step Functions vs Apache Airflow, but the simple guidance might be that Step Functions is a fully AWS-native (and serverless) orchestration engine. Whereas, of course Apache Airflow is an open source project with a diverse ecosystem of other plugins.
I continue to be confused about AWS offerings other than their EC2 and data storage (S3/Redshift/Spectrum/Aurora) solutions which are undoubtedly amazing products. The thing with Workflow scheduling and orchestration is that it's complex and non trivial. I'd rather buy a product built by a company with a focus in this area (i.e. Prefect) rather than go with yet another poorly executed but highly marketed AWS product. When I buy a product for my team I buy support and integration first, technology is less important. A lot of AWS products are poorly supported and mostly integrate with other AWS stuff. On top of all of this we are supposed to be running containers and AWS makes all this serverless stuff which defeats the whole purpose of moving applications to containers smoothly running in the cloud.
67 comments
[ 2.4 ms ] story [ 122 ms ] threaddisclosure: co-founder of astronomer
"Environments with CREATING status must complete previous operation before initiating a new operation."
can't email support for help (i only have basic plan)
any AWSMWAA ppl on this thread and can help? the instance name is `airflow-ry-test`.
Is this a good idea? I don’t think so
Or people working to make some *nix tools work on different distributions.
Alas, this relies on many things, not the least of which are peering agreements between cloud vendors that do not punish the consumer for using a service on one cloud with services from another.
It is possible to an extent today, but for competitive reasons, cloud companies do not seem to have a built in incentive to collaborate. Perhaps the cost of switching and lock-in will force cloud providers to work together in the long one, else they risk alienating customers, and then lack the ability to capture a larger market.
In other words, competition is good as long as people can realistically take advantage of it. I think cloud vendors have an obligation to see that this competition is encouraged and supported.
I agree.
It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge it, and use it - to the exclusion of all other implementations - not because of bias, but because of the truth underlying our meritocracy.
The question for me is - when we engage in said meritocracy, with all these BDFLs calling the shots, will they be the ones to negotiate with Jeff, Bill, and Larry? Or will we?
but are largely not, narrowing to sharing core tweaks, and keeping the PaaS layers proprietary
so it's not the maintainers failing to pick & unify, but Google etc employees not giving back
Of course their storage backend is going to be S3 and they are going to send logs to Cloudwatch, we have been doing it in a similar way for quite some time and it is what I expect from a solution managed by AWS.
Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
https://github.com/spotify/luigi
We’ve been using it for complex update workflows for about 5 yrs now, and it just works.
It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
I would be also curious to hear more about alternative and mature open-source solutions to Airflow.
[1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
Disclosure: I'm on the Airflow PMC.
Also, how would you describe the overall health of the code base with all of these new features added?
Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video.
This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results
Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...
In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.
I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.
Disclaimer: I'm the creator.
I realize you are buying not just the compute, but the management, but that ends up being something in the cost range of $300-$500 or so per month for the airflow management part of it. Seems a bit steep. $50-$100/mo would be a no brainer for us. For some orgs I can see this being a great solution, but its not really friendly for the little guy (with a min price of $350/mo).
I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ against those outages.
I used to manage an airflow deployment (of which my team was the primary consumer), and it was not enjoyable in the least.
For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely.
But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.
I am really just surprised it is not a "no brainier price". It's a tough sell. I am sure it's valuable for some people, I just can't really justify using it. I think value to me is like $100/mo (as random airflow user on the internet).
They can certainly sell this for what they want, I just won't buy at this price.
edit: in this sense...i know the fundamental differences otherwise
edit #2: we're also not a SaaS, but a B2C
So far it's been very low maintenance - outside of the few random scares where dag logs filled up my server - but then I researched and found maintenance Dags that prevent a lot of issues.
Wondering when my next outage will be is always fun, but it's been pretty stable so far.
E: I know and appreciate the tech they put into it. It's just too high of a price for me once I get the workers added. I still want migrate mine to fargate workers at some point though.
It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.
https://cloud.google.com/composer/
AWS: “we will build one in 3 months”
See. nobody care about the differentiator in big 2B enterprise business . It’s more about trust and migration cost
I don't think of Glue as a generic workflow system the way the others are - it's definitely much more optimized for ETL use cases.
With time, I'm sure there'll be more detailed guidance on Step Functions vs Apache Airflow, but the simple guidance might be that Step Functions is a fully AWS-native (and serverless) orchestration engine. Whereas, of course Apache Airflow is an open source project with a diverse ecosystem of other plugins.
[1] https://aws.amazon.com/swf/