> With Terraform, your local executable makes rest calls to each service’s REST API for you, meaning no intermediary sits between you and the service you’re controlling. Want an RDS instance? Terraform will make calls directly to the RDS API.
How is this different than CloudFormation making the same calls?
You give CloudFormation a list of instructions. It accepts it and gives you an ID to watch for updates, then it goes off and executes them.
Terraform executes a list of instructions. It executes them in front of you while you wait.
Both are fine until you run into something like this:
I'm pushing a Elastic Container Service Task Definition change via CDK. A CloudFormation change is submitted, and I wait for it to finish. In the background, it's trying to do the update but the update fails due to some misconfiguration with the new container.
CloudFormation doesn't fail or return an error. It times out after an hour and reverts the change. I have to know to dig into the AWS console to find my failed tasks to view the error.
If I did this update via Terraform, I would get the error back in my console quickly as Terraform is directly telling ECS to make the change. With CDK, the CloudFormation changeset is generated, it is submitted to CloudFormation, then the tool polls the AWS API for progress updates. Sometimes you get specific messages back, sometimes it fails and you need to go in and see what it failed on.
That's right - use AWS CDK instead. You don't have to worry about the low-level CloudFormation syntax and details. I switched a few years ago and haven't looked back. CDK keeps getting better and better, also handling things like asset deployments (Docker images, S3 content, bundling), quick Lambda updates with --hotswap, quick stack debugging with --no-rollback, etc.
I agree, and have the same experience. CDK is so much easier, much less verbose, and unit testable (at least to some degree).
Since resource importing is possible in CDK (not nice, but possible) you can even start using it if you already have resources that you do not want to recreate.
I'm always surprised that more people arent aware of CDK. Its an extremely powerful way to write software. Especially once you get good at it. CFN pales in comparison, CDK to me feels like the future of software development.
Maybe CloudFormation is too entrenched at the moment? I think CDK became publicly available in late 2019 or early 2020.
I haven't used it because I'm currently working on a project using Serverless (thin layer on top of CloudFormation) with no easy way to slip any CDK in.
My comments are my own and don't represent my employer.
Almost every new service built inside Amazon uses CDK. I too am surprised that more people aren't aware of it. And you're right once you get good at it you can spin up infrastructure incredibly quickly with minimal guess work.
Yea the nodejs dependency is an abomination. The cdk also requires a nodejs version newer than what comes on many linux distributions so you end up installing a custom nodejs version just to run cdk.
I don’t know why they picked nodejs over a more sensible language such as python3 which comes installed almost anywhere or golang which is easy to distribute in binary form and relatively reverse compatible.
The aws cli really got it right: just let me install a standalone binary and call it a day.
AWS CDK also works with Python. However, in my experience, TypeScript-based CDK projects are more productive to develop, due to all the built-in error checking and autocompletion features in Visual Studio Code. Node.js also provides a more coherent environment with the package.json structure, which defines all the scripts and dependencies etc. Python-based CDK projects seem to include more "custom hacks". Because of all this, I prefer to use TypeScript for CDK even if the actual application to deploy is based on Python.
Regardless of what language you use to write your cdk stacks, the cdk cli application which transpiles your cdk stacks into cloudformation requires nodejs. So in reality cdk requires nodejs — always — and then whatever language you are writing the stacks in as well. It becomes a pain in the ass when you have a python application which is defined in a python cdk stack, but you still need nodejs to transpile your python cdk stack into cloudformation.
My main gripe is that the cdk cli should be distributed as a standalone binary so I don’t need to install nodejs to deploy my project.
Ok thanks for clarifying. I guess we already have Node.js everywhere where CDK is used, so never noticed it. I can see it's uncomfortable when that is not the case.
That's very true. :-( I really don't understand why AWS doesn't optimize the CloudFormation backend. I don't see any reason why it couldn't update resources basically as fast as the APIs allow.
The purpose of CDK is exactly to reduce the complexity of your IaC. It automatically sets up all kinds of defaults, such as required permissions between different cloud resources that need to talk to each other. The underlying complex things are handled by AWS-managed library code.
This doesn't even include one of the most useful features of CDK: truly modular components. I can create a module used 100 times and all you have to worry about is the constructor and inputs instead of the nightmarish lack of reusability in other IaC.
CDK is terrible and way too low level for most apps, and worst of both worlds for code/declarative IaC (verbose & cognitive overhead heavy way to try to author code that generates a CF template - that still may not work).
Use AWS Copilot, it's like a reboot of Elastic Beanstalk complete with good CLI, made for the Fargate age and lets you extend it with bits of CF. (https://aws.github.io/copilot-cli/)
It sounds like your development is focused on containers, and perhaps Copilot is best for that. AWS CDK works very well when your application is based on many different kinds of cloud resources that need to be connected together, utilizing all the serverless and managed features of AWS.
Copilot supports serverless too (App Runner, serverless Aurora, DynamoDB, S3, CodePipeline etc). But if your app is using a huge nr of AWS features, you're probably over engineering it.
Sure it's opinionated enough that it won't fit every use case people have on AWS, but in most cases for new apps it's good.
I've developed cdk for about 6 months now and it seems to me like a great idea.
However, the implementation is quite lackluster.
- It's very slow. Deploying our app from scratch takes >30 minues. An update of a single lambda (implementation, not permissions) takes about 3-5 minutes.
- Bad error information. On an error, cdk usually just reports which component couldn't be created. Not why it couldn't be created. That information you 'might' find in the aws cloudformation console, or not.
- It's buggy. Some deployments randomly fail because cdk couldn't find your cli credentials. Sometimes resources can't be created. In each case it works if you deploy again...
I suspect that most shortcomings that I experience are due to cloudformation. I'd really be interested in a 2.0 version that is either highly optimized or polished, or works with a different intermediate abstraction entirely. Updating and validating a tree of resources and policies should not be that difficult, at least not for AWS.
I think that the --hotswap option does what you are hoping for speedier updates. It allows CDK to update some cloud resources directly (e.g. Lambda function code) so that they "drift" away from the CloudFormation template definition. It's good for quick development.
CloudFormation, with its HORRIBLE YAML templating (whatever dsl/language) and arcane error messages is a horror story. I hate it so much that I'd rather quit my job than debug why CloudFormation decided for no reason to update my RDS instance for a PR that was just a README file update.
To be honest, I don't agree this. Manage an infrastructure need evidence and trace how this get created. I've been in the situation a few times. Have been threw projects terraform code doesn't match aws infrastructure. We don't know when an how the drift happen. At least, cloudformation can have some feature to detect the difference and help me trace back which commit actually has been deployed. CDK make the job easier for developers because it deliver some convenience and offer more pattern to write code. I like both.
Vanilla cloudformation is bad, but so is terraform (for my use case anyway). We wrap our cloudformation with python, you need something similar for terraform to make it less terrible (cdktf, terragrunt, terrascript).
Managed services offer big benefits over software. With CF, new stacks, change sets, updates, rollbacks and drift detection are an API call away.
Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away.
Using a single cloud provider has a big benefit over a multi-cloud tooling. I only run workloads on AWS, so the CF syntax, specs and docs unlocks endless first party features. A portable Terraform + Kubernetes contraption is a lowest common denominator approach.
Of course everything depends.
I've configured literally 1000s of systems with CloudFormation with very few problems.
I have seen Terraform turn into a tire-fire of migrations from state files to Terraform enterprise to Atlantis that took an entire DevOps team to care for.
Agree. CF is not a magic bullet, but neither is ansible or terraform.
We used ansible heavily with AWS for 2 years. Then we decided to gut it out and do CF directly. Why? If we want to switch clouds, it's not like the ansible or terraform modules are transferable ... So might as well go the native supported route.
I agree with the article, messages can be cryptic, but at the end of the day, I have a CF stack that represents an entity. I can blow away the stack, and if there's any failure or issue, I can escalate my permissions and kill it again. Still a problem? Then it's AWS's fault and a ticket away (though I've only had to do this once in 5 years and > 150,000 CF stacks.
I also would argue, if a stack deletion stalls development, you are probably using hard-coded stack names, which isn't wise. Throw in a "random" value like a commit or pipeline identifier.
I've had far less issues with CF than terraform or ansible. I have yet to see CF break backward compatibility, while I had a nightmare day when I couldn't run any playbooks in ansible because the module had a new required parameter on a minor or patch version bump.l (which was when I called it quits on ansible, I then relooked at terraform, and decided to go native)
I will caveat that our use case for AWS involves LOTS of creation and deletion, so I find it super helpful to manage my infrastructure in "stacks" that are created and deleted as a unit.. I dont need to worry about partial creations or deletions.. like ever... It basically never fails redoing known-working stuff... Only "first time" and usually because we follow least-privilege heavily
Yes Ansible does have extensions and can be used to provision AWS services.
The approach between Cloudformation/Terraform/Pulumi and Ansible are entirely different though.
The former are declarative, they define how the end state should look. Ansible is a task runner, you define a set of manual tasks it needs to execute to get to the end state.
I strongly advice against using Ansible for provisioning resources. It's idempotent by convention only. When I had to reluctantly use it for jobs it was extremely difficult to get a repeatable deterministic environment set up. Each execution lead to a different state and was just a nightmare to deal with.
Cloudformation/Terraform/Pulumi are much better in that regard as they generate a graph of what the end state should be, check the current state, generate an execution plan how to make the current state look like the target state.
Where Ansible is better than Cloudformation/Terraform/Pulumi is you have a bunch of servers already set up and you want to change the software installed/configuration on them. That's bit of an anti pattern these days changing config/provisioning at runtime. You can change that slightly and use Ansible with Packer to generate pre-baked images which works ok if you don't mind lots of yaml. This isn't to bad and works reasonably well and works to Ansible strengths all though these days most people don't prebake images with containerization. Also if you are only using Ansible for provisioning config on a host Nix achieves this much more elegantly / reliably.
We already used ansible for other things, so it wasn't too hard to swap over to AWS modules... (Except they were inconsistent and poorly supported, we ultimately found out)
Someone at Hashicorp then convinced mgmt that terraform is almost a write-once system, and we could jump from AWS to Azure or GCP easily "just change the module!"... When actual engineers looked at it, after 3 days there was almost a mutiny and we rejected terraform mostly based on the fact someone lied to our managers to try and get us to adopt it... I know someone who is very happy with terraform nowadays, but that ship sailed for us.
Those were basically the only people in this space, so we started rewriting ansible to CloudFormation. Since we mostly use lambdas to trigger the creation of CF stacks, this really works well for us, since our lambdas exist for less than a second to execute, and then we can check in later to see if there's issues (which is less than 1 in 50,000? 100,000? in my experience... Except for core AWS outages which are irrespective of CF). Compared to our ansible (and limited terraform) setups which required us to run servers or ECS tasks to manage the deploy. We can currently auto-scale at lambda scale-up speed to create up to 30 stacks a second if demand surges (the stack might take 2-3 minutes to be ready, but it's now async). Under ansible/terraform we had to make more servers and worker nodes to watch the processes... And our deployment was .3/.4 stacks per minute per worker (and scaling up required us to make more workers before we could scale up for incoming requests)
If I was building today, I'd probably revisit terraform, but I think the cdk or CF are still what I'd recommend unless there's a need for more-than-AWS... E.g. multi-cloud deployments, or doing post-creation steps that can't be passed in by userdata / cloud-init.. in which case CF can't do the job alone and might not be the right tool.
I'm a big proponent of CF when you are using AWS, but if you are on GCP, don't even bother with their managed tool, just go straight to TF. Their Deployment Manager is very buggy (or at least it was 2 years ago).
CloudFormation/Terraform/etc are also configuration management programs. They just work on the APIs of cloud vendors, rather than a package management tool's command-line options. They've been given a new name because people want to believe they're not just re-inventing the wheel, or that operating on cloud resources makes their tool magically superior.
> We used ansible heavily with AWS for 2 years. Then we decided to gut it out and do CF directly. Why? If we want to switch clouds, it's not like the ansible or terraform modules are transferable ... So might as well go the native supported route.
>
> I agree with the article, messages can be cryptic, but at the end of the day, I have a CF stack that represents an entity. I can blow away the stack, and if there's any failure or issue, I can escalate my permissions and kill it again. Still a problem? Then it's AWS's fault and a ticket away (though I've only had to do this once in 5 years and > 150,000 CF stacks.
Another killer feature is StackSet. I managed to rewrite datadog integration CF (their version required manual steps) to a template that contained custom resources that made calls to DD to do registration on their side.
I then deployed such template through StackSets and bam, every account in specific OU automatically configures itself without any manual steps.
> Managed services offer big benefits over software. With CF, new stacks, change sets, updates, rollbacks and drift detection are an API call away.
>
> Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away.
The problem is when those help tickets get responses like “try deleting everything by hand and see if it recreates without an error next time”. They've worked on CloudFormation over the last year or but everyone I've known who's switched to tools like Terraform did so after getting tired of unpredictable deployment times or hitting the many cases where CloudFormation gets itself into an irrecoverable state. I can count on no fingers the number of development teams who used CF and didn't ask for help recovering from an error state in CF which required out-of-band remediation.
I believe they've also gotten better at tracking new AWS features but there were multiple cases where using Terraform got you the ability to use a feature 6+ months ahead of CF.
> A portable Terraform + Kubernetes contraption is a lowest common denominator approach.
Terraform is much, much richer than CloudFormation so I'd compare it to CDK (with the usual aesthetic debate over declarative vs. procedural models) and it doesn't really make sense to call it LCD in the same way that you might use that to describe Kubernetes because it's not trying to build an abstraction which covers up the underlying platform details. Most of the Terraform I've written controls AWS but there's a significant value to also being able to use the same tool to control GCP, GitLab, Cloudflare, Docker, various enterprise tools, etc. with full access to native functionality.
Terraform (and kubernetes) itself aren't a lowest common denominator, however I believe the comment alludes to an approach where you try to abstract cloud features. This can (kind of) reasonably be done with terraform and kubernetes and avoiding vendor specific services such as various ML services, DynamoDB, etc.
However, you can use terraform just fine while still leveraging vendor specific services that actually offer added value, like DynamoDB or Lambda. Cloudformation however doesn't really offer that much added value (if any) over terraform, so using terraform isn't an LCD approach perse.
Yes — that's basically what I was thinking: you could make an argument that using Kubernetes inherently adds an abstraction layer which might not be preferable to using platform-native components but it sounded like the person I was responding to was making the argument that using Terraform requires that approach.
I found that especially puzzling because one of the reasons why we switched to Terraform was because it let us take advantage of new AWS features on average much faster than CloudFormation.
> Managed services offer big benefits over software.
TF can be used as a managed service.
> Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away.
The same is true with TF, except 100000% better unless you're paying boatloads of money for higher tiered support.
> I only run workloads on AWS, so the CF syntax, specs and docs unlocks endless first party features.
CF syntax is an abomination. Lots of the bounds of CF are dogmatic and unhelpful.
> I have seen Terraform turn into a tire-fire of migrations from state files to Terraform enterprise to Atlantis that took an entire DevOps team to care for.
CF generally takes an entire DevOps team to care for, for any substantial project.
Sure, but I never seen that myself. If TF was used it was always own set up infrastructure at best.
> The same is true with TF, except 100000% better unless you're paying boatloads of money for higher tiered support.
Again, all places I worked had enterprise support and even rep assigned. I think I only used support for CF early on, I don't know if it was buggier back then or I just understood it better and didn't run into issues with it.
> CF syntax is an abomination. Lots of the bounds of CF are dogmatic and unhelpful.
I would agree with you if you were talking about JSON, but since they introduced YAML it is actually better than HCL. One great thing about YAML is that it can be easily generated programmatically without using templates. Things like Troposphere make it even better.
> CF generally takes an entire DevOps team to care for, for any substantial project.
Over nearly 10 years of my experience, I never seen that to be a case. I'm currently in a place that has an interesting approach: you're responsible for deployment of your app, so you can use whatever you want, but you're responsible for it.
So now I'm working with both. And IMO I see a lot of resources that are not being cleaned up (because there's no page like CF has, people often forget to deprovision stuff), also seeing bugs like for example TF needs to be run twice (I think last time I've seen it fail was that it was trying to set tags on a resource that wasn't fully created yet).
There is also situation that CF is just plain better. I mentioned in another comment how I managed to get datadog integration through a single CF file deployed through stackset (this basically ensured that any new account is properly configured). If I would end up using TF for this, I would likely have to write some kind of service that would listen for events from the control tower, whenever a new account was added to OU, then run terraform to configure resources on our side and make API call to DD to configure it to use them.
All I did was to write code that generated CF via troposphere and deploy it to stackset in a master account once.
Right, your post is mostly "I like the thing that I've used, and I do not like the thing I haven't used". They're apples and different apples.
> Again, all places I worked had enterprise support and even rep assigned
So, again, you've worked at places that were deeply invested in CF workflows.
> but since they introduced YAML it is actually better than HCL. One great thing about YAML is that it can be easily generated programmatically without using templates.
Respectfully, this is the first-ever "yaml is good" post I think I've ever seen.
> Over nearly 10 years of my experience, I never seen that to be a case. I'm currently in a place that has an interesting approach: you're responsible for deployment of your app, so you can use whatever you want, but you're responsible for it.
I'd love to hear more about this.
> And IMO I see a lot of resources that are not being cleaned up (because there's no page like CF has, people often forget to deprovision stuff), also seeing bugs like for example TF needs to be run twice (I think last time I've seen it fail was that it was trying to set tags on a resource that wasn't fully created yet).
I guess we're just ignoring CF's rollback failures/delete failures/undeletable resources that require support tickets then?
> There is also situation that CF is just plain better. I mentioned in another comment how I managed to get datadog integration through a single CF file deployed through stackset (this basically ensured that any new account is properly configured). If I would end up using TF for this, I would likely have to write some kind of service that would listen for events from the control tower, whenever a new account was added to OU, then run terraform to configure resources on our side and make API call to DD to configure it to use them.
Again respectfully, yes, the person that both doesn't like and hasn't invested time into using Terraform at scale probably isn't going to find good solutions for complicated problems with it.
While this is true and AWS support is very responsive and useful, it doesn't mean they solve all the problems. Sometimes their help is: "I'll note that as a feature request, in the meantime you can implement this yourself using lambdas".
Main use case of Terraform is not portability. Have fun porting your SQS queue, DynamoDB table or VPC config to GCP or Azure equivalents. It won't look like similar at all except the resource name. However, If you are only running containers and virtual machines, sure you can benefit from portability
Cloudformation lags features so bad that you end up hacks or Lambda functions with custom resources. DynamoDB Global tables took 4/5 years to be available in Cloudformation.
I've also seen wrongly constructed Cloudformation deleting critical databases, hanging (often), times out, rollback hanging not succesful, so it's not always rainbows & sunshine there either. Also I don't like Terraform for its usage of excessive state files, handling state files with DynamoDB locks, having them on S3.
I won't deny the good features of it like being managed is a huge plus, but it's so slow and lagging behind, and it's YAML is so verbose an has stack size limits, it's always a workaround with Cloudformation. My company uses it by abstraction for internal PaaS and deployment automation and it takes a lot for trivial changes to complete.
So in short, neither is perfect, but for me Terraform is easier to use, easier to debug, and faster, and features don't lag nearly as CF. Those are good enough reasons to avoid Cloudformation for me. I also don't like CDK because it's too verbose and its still CF, and I would rather generate Terraform JSONs/HCLs myself if I need more logic either.
Terraform also helps when you need to configure multiple stacks i.e for a service you can have a module that reflects your organizational best practices, a Fargate service for runnning container, automatic Datadog dashboards, cloudwatch alarms connected to Opsgenie/Pagerduty etc.
I really don't get why features come so late to CloudFormation - I guess AWS don't use much CloudFormation internally then, but surely they're not stringing together AWS cli calls? CDK is reasonably new too, unless they waited a long time to go public with it.
CDK is a higher-level (and awesome in my experience) way to just generate CloudFormation specs. In other words, you need both CloudFormation and CDK support for features to become available there.
In terms of getting new features fast CDK is strictly worse than CloudFormation.
Many/most teams internally using AWS use CloudFormation; an AWS service I was a part of was almost entirely built on top of other AWS services, and the standard development mechanism is to use CFN to maintain your infra. You only do drastic things like "stringing CLI calls" if there's something missing from CFN and not coming out soon enough, in which case maybe someone writes a custom CFN resource and you run it in your service account.
Depending on how old the service is, the ownership of the CFN resource may be on the CFN service team (from back when they white-gloved everything) in which case there are massive development bottlenecks (there are campaigns to migrate these to service team ownership) or more often the resource is maintained by the service team itself, in which case the team may not be properly prioritizing CFN support. There can be a whole lot of crap to deal with for releasing a new CFN _resource_, though features within a resource are relatively easy.
On my last team, we did not consider an API feature delivered until it was available in CFN, and we typically turned it around within a couple of weeks of general API availability.
> hanging (often), times out, rollback hanging not succesful
The timeouts in CF are ridiculous. Especially with app deployments. I can't remember which service it is, but something can wait up to 30min on a failed `apply` and then wait the same on a failed revert. Only then you can actually deploy the next version (as long as it wasn't part of the first deploy, then you get to wait until everything gets deleted as well).
(yes, in many cases you can override the timeouts, but let's be honest - who remembers them all on the first run or ever?)
I've been using CF for a few years now with minimal complaints but I just hit a create changeset endless timeout (2 days to finally time out).
The worst part is that there are no error messages. When it fails and I click "Details", it takes me to the stack page and shows 100% green. Support ticket seems slow to get a response too.
That aside, my overall experience has been positive!
Terraform has its fair share of lag as well. One particular case that irks me is that the "meta" field on Vault tokens is unsupported. Vault of course being another first-class Hashicorp product makes this particularly odd.
That being said the Vault provider is open source and it's quite easy to add it and roll your own.
If I'm remembering correctly, I'm pretty sure the Vault provider for Terraform was originally contributed by an outside company rather than inside HashiCorp. My guess would be that it has encountered what seems to be a common fate for software that doesn't cleanly fit into the org chart: HashiCorp can't decide whether the Vault team or the Terraform team ought to own it, and therefore effectively nobody owns it, and so it falls behind.
Pretty sure that's the same problem with CloudFormation, albeit at a different scale: these service teams are providing APIs for this stuff already, so do they really want to spend time writing another layer of API on top? Ideally yes, but when you have deadlines you've gotta prioritize.
Do you understand what is the difference between the things you can express with Dhall or ML vs Go or TS?
People are so hung up because we could do so much better with expressing only valid states and you do not need to deploy your infra to figure out that and S3 bucket cannot have a redirect_only flag set and also a website index document set at the same time.
CloudFormation can be very flexible, especially with tools like sceptre, it can work very well. A huge issue is that WITHOUT tools like sceptre, you can't really use stacks except as dumb silos. You already need additional tooling (sceptre, CDK, SAM, ...) to make CF workable. I think that most people who despise CF haven't got good tooling.
The issue with CloudFormation is that it lags behind all other AWS services quite often. It seems to be maintained by another team. I realize that getting state management for complex inderdependent resources right requires time and diligence, BUT it's a factor in driving adoption.
- New EKS feature? Sorry, no knob in CF for MONTHS.
- New EBS gp3 volume type available? Sorry, our AWS::Another::Service resource does not accept gp3 values for months after feature release.
- A AWS API exposes information that you would like to use as an input to another resource or stack. SURPRISE, CloudFormation does not return that attribute to you, despite it being available. SORRY NOT FIXABLE.
- Try refactoring and moving resources in/out of stacks while preserving state? Welcome to a fresh hell.
- Quality of CloudFormation "modules" varies. AWS::ElasticSearch::Domain used to suck a whole lot, AWS::S3::Bucket as a core service was always very friendly.
- CloudFormation custom resources are a painful way of "extending" CF to suite your needs. So painful that I refuse to pay the cost of AWS not keeping their stuff up to date and well integrated.
These kinds of lag, this kind of incompleteness when it comes to making information from AWS APIs available have driven me to Terraform for things that are best done in Terraform and require flexibility and CloudFormation for things that work well with it.
At the end of the day CF is a closed-source flavour of Terraform + AWS provider. I would like have gone all in, but just doesn't work and costs hacks, time and flexibility.
That being said, if you have no idea how to work with TF, tell the devs to use CF.
The experience of AWS support is probably also very different when it comes to feature requests. An org that spends half a billion with AWS will get their requests implemented ASAP whereas small fish have to swim with the flow and hope it works for them.
I'm pretty sure that was sarcasm. I disagree with said sarcasm, because CDK takes you one layer away from the actual thing that gets run but gives you a much nicer thing to work with so it can still be a good trade off; writing rust (or whatever) "obfuscates" the underlying CPU instructions but it still turns out to be a good idea.
CloudFormation is great because of its transactionality, so it lends itself nicely to deploying multiple services which are versioned together. You either succeed fully, or all services will be rolled back.
This way you can deploy your whole infra with Terraform, and then deploy to your i.e. ECS cluster using CloudFormation. Works great in practice.
The rollback functionality of CF is a blessing. We use both CF and Terraform at my company and i vividly recall multiple times where my connection had cut out during "terraform apply" and left the Terraform infrastructure in a half-finished state.
Rollback doesn't always work with CF. I've noticed so many times that it would mostly delete everything but not certain things once in a while. Then you're left having to play detective to manually figure out what you need to delete while having to delete dependencies by hand in a specific order.
I've spent hours just waiting for CF to fail deleting EKS or RDS related resources then I end up getting billed for $30+ a month sometimes because I forgot to manually delete a NAT gateway.
When it works, which is a big caveat: we had far more cases where it failed in a way which required manual remediation and the gaps in validation meant that you'd be in a “apply / error / rollback” loop requiring 20+ minutes before you could try again. Terraform was always considerably faster but it was especially the orders of magnitude improvement in retry time which convinced most of us to switch.
The CloudFormation team has been working on this so it's possible that experience has improved but the scar tissue will take time to fade.
It's especially interesting from a product design standpoint: everyone knows things break sometimes but when it forces you out of an easy path into something much harder or simply locks you into a lengthy penalty time delay, people will remember that one time FAR more vividly than all of the times it worked normally.
This is especially true for things like CloudFormation where the user is likely on a deadline or in a stressful situation trying to fix a problem or hit a deployment window. Forcing someone to wait an unpredictable amount of time for no reason ramps up the stress level massively.
Unless things have changed in the meantime, the killer feature of CloudFormation for me is that I don't have to keep track of the state locally. Having to set up tracking of the infra state in Terraform is a huge pain, since it should be stored independently of both the infra code (to allow deploying anything but HEAD) and the infra itself (duh). As long as Terraform doesn't query the existing infra to work out what needs doing I don't want to go back to it.
While the read was interesting and informative, something about the tone made me search for a disclaimer/disclosure of interest. Are you an "influencer?"
I spent a bit of time trying to deploy a lambda app with Cloudformation. I wanted to use a relational database, so I needed to handle migrations.
Ok, so apparently I need to write a custom Cloudformation resource to execute a lambda function that will run the migrations prior to deploying the new version of the lambda. Kind of neat that you can do that.
Except I messed up the output of the custom resource lambda and Cloudformation completely locked my deployment up for 3 hours. 3 hours. I couldn't do _anything_ - rollback, update, whatever.
Cloudformation via a CDK is interesting, and I don't hate it, but oh boy if it gets into a weird state it can completely kill your iteration loop. And the docs say something along the lines of "if it's stuck for too long contact support". No thanks.
CF does have a lot of quirks (especially stacks locking up for various reasons, or rollbacks taking hours).
I find it easiest to run migrations when an application is first starting up (with an appropriate transaction lock so other instances won't cause the migration to run more than once), this way you don't have to do a lot of devops magic for it to work.
That’s not a fantastic model, especially for short lived lambdas. In some small environments just running a migration check on boot is passable, especially if you don’t have better tooling to handle this, but I don’t want to double my execution time working out if I need to run a migration or not in a lambda.
I guess I could do the migration check in the lambda startup rather than in the request path, but it’s still not ideal especially when handling rollbacks.
I might have preferred to use SAM, which uses CF under the hood but is much more focused on getting lambdas to work.
I'd pass in DB settings as parameters to the SAM file, rendering them as env. variables for the lambdas. with DB access managed by IAM roles (so you don't need sensitive info in the lambda env).
And depending on size/scope of project/team, might have one CF for the DB infra, and another for the app lambdas.
Using IaC services means I can have both dev and prod infra, so I can test DB migrations.
And I can do this as a solo dev on a side project where I'm also responsible for front-end and the other back-end stuff plus keeping clients happy on evenings/weekends.
Something funny (well, kind of sad) about CloudFormation I noticed this summer was that if you deploy a CloudFormation stack which updates a ECS service and deploys tasks which then fail health checks, CloudFormation will do nothing about this and just let ECS keep killing and restarting tasks for.. well, at least several hours. You have to know to go into ECS and drain the tasks manually and then initiate a rollback from CF to get your service back into a good state. The bug reports about this I found were going back years.
The upside is that I got really well acquainted with how ECS worked.
100x this. Prior company committed to doing Infrastructure as Code and CloudFormation worked well except for this hiccup. We didn't even have that many services on ECS but we probably had 1 ticket a week asking support to help us with a 'stuck' stack.
Our commitment to CloudFormation was doubled down on that we could do containers, Lambda, and 95% of any other AWS Services....
However, in hidsight using SAM and the ECS CLI probably would have resulted in a more predictable CI/CD process as we weren't fighting deploy semantics through CloudFormation abstraction.
What we ended up doing is mostly decoupling CF and tasks. CF manages the infrastructure. Software deployments / tasks are done by another system which understands issues like this one.
This support for CloudFormation across AWS services is crazy bad. For the longest time you couldn't have an ALB rules with it. To fill this gap you have to create dreaded custom resources... which can get suck for 4 hours.
IIRC, the problem was that rollbacks couldn’t start until it agreed that the service deployment had failed, which wouldn’t happen until I manually drained the task (or waited an unknown period for CF to timeout). You could click the rollback button but nothing would happen. I don’t think the alarm would’ve helped, but maybe I’m misunderstanding.
CFn will allow you to specify up to 5 alarms to monitor during stack deployment as well as a duration to monitor them for up to 3 hours.
During deployment, the stack does not transition into the COMPLETE state until the monitor duration has completed without the alarms being triggered. If one of the alarms is triggered, the stack immediately rolls back.
Since failed ECS tasks create CW Events, you can set up alarms on them and use them as the rollback monitor trigger.
Your updating the service to a broken image? I would say ensuring it's healthy falls outside of the scope of CF responsibilities. `aws ecs update-service` won't even do that. ECS should wait till your new image is healthy before draining previous image versions. If you deployed a bad image, probably needs someone to look at it anyway.
Yes! It drove me nuts! I had a task that I forgot to give a / 200 ok and it just thrashed like crazy. I think once I have the cluster auto rollback it solved the issue though (other than ensuring a reliable health check). They should fix this!
Writing Terraform scripts for AWS is 70% of my job. I do have some issues with the AWS provider in Terraform. Firstly, there are bugs. I ran into a bug a few days ago where the ARN attribute on a Lamba alias was resolving to the ARN of the Lambda, not it’s alias. I only figured it out because I found a GitHub Issue. Additionally, Hashicorp is often playing catch-up with Amazon. A few days ago AWS released a new instruction set architecture for Lambdas that would save my org a lot of money. However after I saw the announcement in AWS I see tons of different GitHub issues created to add this functionality. So I start editing my files based off the documentation only for that issue to be closed and pointed to a new one with different syntax. So I start working off the new syntax only for that issue to close and be pointed to a different one
I've seen this argument everywhere. If this is the web, people would say "use the platform", because aparently any abstraction made on top of that is automatically worse, if this is .Net it would be "use Azure". If this is desktop applications... use C++?
That's just not how reality works. First class citizens can suck, because guess what, they're designed by mortals
not the case with cdk. i hate IAC in general. terraform being one of my least favorites due to mostly being declarative, verbose, and the way it manages state. cdk is a large improvement and moves towards just being code.
Their service owners don’t seem to feel that way. Terraform’s AWS provider often adds support for new services and features weeks or months before CloudFormation does.
One of the most amazing things I saw at AWS reInvent was an advanced talk on IaC that provided the code of a lambda function inline in a CloudFormation template. I realize that this is just one talk, and there are plenty of ways to structure things well, but this practice is directly encouraged by the design of CloudFormation[1]. AWS has attempted redefining the lambda deployment story multiple times, there are multiple companies whose primary offering is providing a better way to deploy code to serverless offerings, but this still stands out to me as one of the most terrible ways to do things, and I blame the design of CloudFormation.
I'm going off track here but Pulumi have a totally mind-bending feature where you can write the code of a lambda function not only inline, but such that it captures the value of variables from the surrounding infra code at the time the function is serialized.
Seeing the specific examples they use it for (AWS infra glue) makes me think that there is room for infrastructure related lambdas to be defined right in cfn or infra code, with very low ceremony, even if you wouldn't want to deploy "applications" like that.
I'm working with what I think is a similar problem-space, trying to create a Node.js program that produces code to be run on the browser. The code runs in Node.js at "build time" which essentially means constructing the static web-site, which means some JavaScript must be somehow "deployed" to the website to be then later run at "run time".
So deploying to an AWS etc. Lambda seems conceptually similar to deploying from static-site-generator to a static web-site source-code, which is then what the browser will execute. Or is it?
Can we just agree that none of these tools (Terraform, CloudFormation, CDK) are particularly good? None of them really achieve the goal of purely declarative, immutable, stateless infrastructure-as-code and all of them have significant disadvantages.
Agreed. We use Pulumi and it's fine, but the state-managing feels brittle.
I think the state-of-the-art is Kubernetes + jsonnet, though you still need to suffer through the learning curve and you'll still need TF and company to provision some of your cloud resources.
The C in IaC stands for Code, and code is best written in proper programming languages. Both CF and Terraform are greatly limited by their use of yaml and hcl. Tools like AWS CDK and Pulumi, especially when used with a typed language like TS, are the future of IaC.
you are arguing about another thing, Teraform also has a CDK. But the question isn't whether to use CDK, it's "native" vs third party (Terraform & Pulumi)
I came here to say that Pulumi is the answer, but people already mentioned it. It's TF under the hood, some of the docs links even refer to the links in TF documentation [not intentionally, just by copy and paste], but they made the experience so much better. Writing code in Typescript and seeing your infrastructure provisioned with all the goodies Pulumi added on top of TF is pure joy. No more HCL, or other "here's my DSL, learn it" scenarios.
The Pulumi providers for major clouds (AWS/GCP/Azure) have native versions instead of using the TF providers under the hood. Caveat: some are still in preview.
Pulumi is not "Terraform under the hood" in any sense than that some of the providers are forks of Terraform providers.
The engine is entirely separate and, now that they have a foothold with it, they are building out providers that are built around the Pulumi execution model first-class, rather than an adapter layer.
One notable core difference in the Pulumi model is that it doesn't have "plan" in the same sense that Terraform does, where the set of actions is fixed and just executed once you accept it, returning an error if that turns out not to be possible.
"pulumi preview" just runs the program in a dry-run mode, so there's no guarantee that the subsequent apply will match unless you've been careful to write a totally deterministic program that always behaves the same with placeholder values as it does for concrete values.
That's a typical tradeoff for using a DSL vs. a general-purpose language though, and one that seems to pay off for a lot of teams.
Commenting as an employee, with the account created for that purpose? It's not true what you're saying, but it's good that you're going that route. As I've said, I am a user of Pulumi, and you're doing loads of things good, just don't try to discard that you started as "TF under the hood".
I don't and never have worked at Pulumi, but I've read the open source code and can see that the Pulumi engine is distinct from what HashiCorp seems to call "Terraform CLI" or "Terraform Core".
But there are indeed repositories in their GitHub org which started as forks of Terraform providers, as I said. Pretty good strategy to start from all of the generic glue code somebody else already wrote and then adapt it to a new engine, because that's the undifferentiated heavy lifting that needs to be done regardless of the engine design.
I’ve seen teammates write enough impenetrable JS/Python/ etc code that I’m pretty comfortable saying I want none of that near the machinery that provisions potentially expensive amounts of infrastructure.
I like that HCL is declarative and deliberately restricted in what’s possible, it means I never have to think about some dev writing “clever” code or over abstracting things, or writing bugs that spin up to many machines.
HCL being so restricted is the reason why many teams feel the need to implement a layer on top of it.
Once you’re working with HCL templated via Gomplate or what have you, and a pipeline which processes the code base and then spits out the resultant HCL as an artifact to be used by a subsequent step, you’ll pine for the impenetrable JS or Python code. At least that is code which has a reasonable development workflow associated with it, not a monstrosity held together with duct tape and vulgar hacks.
I recently did a comparison of Pulumi and CDK. Pulumi's AWS abstractions are sadly nowhere near as good as CDK patterns, resulting in far more code needing to be written for the same result. I've also noticed that Pulumi seems to have a hard time accurately tracking state for AWS resources.
Pulumi does work very well in the Azure space as Azure resources are already well abstracted.
I personally prefer aws-cdk over Terraform for the specific use case of writing complex Step Function State Machines. Using code to link and chain steps and logic together is much easier than HCL imo. For everything else I prefer Terraform though.
I find it mildly disturbing to see how willing people seem to be at a personal level to attach themselves skill-wise to single-vendor solutions.
I get that CloudFormation may be a bit better than Terraform in this way or that way. But I have no personal interest in learning it. All that investment of time and effort and when I need to do a project on a different vendor I have nothing, no skills. What's the point?
To be clear: I understand why companies and even specific teams or projects may choose to go all in on vendor-lock-in. It's often the most direct line to the business value they want and as people point out, it's rare to never that most cloud native applications will shift cloud vendors, and even if they do most of a Terraform setup will have to be reworked.
So I get why companies and managers want that. I just don't understand why software engineers in those teams are so happy to go along.
I don’t see how you couldn’t learn both. Terraform is dead simple to learn and cloud formation is the best way to define your infrastructure in code in AWS. Even more wrapped by CDK. I don’t think you would go that far career wise knowing only with Terraform for IaC.
I also don’t understand those organisations that use every single AWS managed solution (RDS, Redis, Elasticsearch, EKS) but then they feel the need to manage their infrastructure state themselves with Terraform instead of choosing the managed solution for this.
As a former AWS employee, Terraform is MUCH better at scale for AWS resources than Cloudformation or even AWS CDK. CDKs reliance on CloudFormation underneath means being limited to the exact same problems that are mentioned in the article, as well as managing resources at enormous scale incredibly painful.
Often, these single-vendor technologies are thin syntactic and nomenclaturial veneers over actually-quite-general abstractions. E.g. it's very easy, if you want or if you need, to rewrite a CloudFormation stack as a Terraform project, or a even a set of Kubernetes custom-resource-controller resource manifests. They're all analogous technologies; the and the "hard part" of an implementation in any of them isn't to do with the things that make them different, but rather the things they share in common — the formal modelling of a digraph of dependent declarative infrastructure resources.
225 comments
[ 0.22 ms ] story [ 281 ms ] threade: Title was changed
How is this different than CloudFormation making the same calls?
Terraform executes a list of instructions. It executes them in front of you while you wait.
Both are fine until you run into something like this:
I'm pushing a Elastic Container Service Task Definition change via CDK. A CloudFormation change is submitted, and I wait for it to finish. In the background, it's trying to do the update but the update fails due to some misconfiguration with the new container.
CloudFormation doesn't fail or return an error. It times out after an hour and reverts the change. I have to know to dig into the AWS console to find my failed tasks to view the error.
If I did this update via Terraform, I would get the error back in my console quickly as Terraform is directly telling ECS to make the change. With CDK, the CloudFormation changeset is generated, it is submitted to CloudFormation, then the tool polls the AWS API for progress updates. Sometimes you get specific messages back, sometimes it fails and you need to go in and see what it failed on.
Since resource importing is possible in CDK (not nice, but possible) you can even start using it if you already have resources that you do not want to recreate.
Almost every new service built inside Amazon uses CDK. I too am surprised that more people aren't aware of it. And you're right once you get good at it you can spin up infrastructure incredibly quickly with minimal guess work.
I don’t know why they picked nodejs over a more sensible language such as python3 which comes installed almost anywhere or golang which is easy to distribute in binary form and relatively reverse compatible.
The aws cli really got it right: just let me install a standalone binary and call it a day.
My main gripe is that the cdk cli should be distributed as a standalone binary so I don’t need to install nodejs to deploy my project.
It used to have a lot of rough edges, but that's (mostly) not the case anymore.
Complexity management is an often overlooked parameter of reliable infrastructure.
Use AWS Copilot, it's like a reboot of Elastic Beanstalk complete with good CLI, made for the Fargate age and lets you extend it with bits of CF. (https://aws.github.io/copilot-cli/)
Sure it's opinionated enough that it won't fit every use case people have on AWS, but in most cases for new apps it's good.
However, the implementation is quite lackluster.
- It's very slow. Deploying our app from scratch takes >30 minues. An update of a single lambda (implementation, not permissions) takes about 3-5 minutes.
- Bad error information. On an error, cdk usually just reports which component couldn't be created. Not why it couldn't be created. That information you 'might' find in the aws cloudformation console, or not.
- It's buggy. Some deployments randomly fail because cdk couldn't find your cli credentials. Sometimes resources can't be created. In each case it works if you deploy again...
I suspect that most shortcomings that I experience are due to cloudformation. I'd really be interested in a 2.0 version that is either highly optimized or polished, or works with a different intermediate abstraction entirely. Updating and validating a tree of resources and policies should not be that difficult, at least not for AWS.
1: https://www.pulumi.com/docs/guides/adopting/from_aws/
Managed services offer big benefits over software. With CF, new stacks, change sets, updates, rollbacks and drift detection are an API call away.
Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away.
Using a single cloud provider has a big benefit over a multi-cloud tooling. I only run workloads on AWS, so the CF syntax, specs and docs unlocks endless first party features. A portable Terraform + Kubernetes contraption is a lowest common denominator approach.
Of course everything depends.
I've configured literally 1000s of systems with CloudFormation with very few problems.
I have seen Terraform turn into a tire-fire of migrations from state files to Terraform enterprise to Atlantis that took an entire DevOps team to care for.
This is a great way of saying "I've never used CloudFormation" without stating it directly.
Hacker News is not a site for snarky accusations of deception.
https://news.ycombinator.com/newsguidelines.html
We used ansible heavily with AWS for 2 years. Then we decided to gut it out and do CF directly. Why? If we want to switch clouds, it's not like the ansible or terraform modules are transferable ... So might as well go the native supported route.
I agree with the article, messages can be cryptic, but at the end of the day, I have a CF stack that represents an entity. I can blow away the stack, and if there's any failure or issue, I can escalate my permissions and kill it again. Still a problem? Then it's AWS's fault and a ticket away (though I've only had to do this once in 5 years and > 150,000 CF stacks.
I also would argue, if a stack deletion stalls development, you are probably using hard-coded stack names, which isn't wise. Throw in a "random" value like a commit or pipeline identifier.
I've had far less issues with CF than terraform or ansible. I have yet to see CF break backward compatibility, while I had a nightmare day when I couldn't run any playbooks in ansible because the module had a new required parameter on a minor or patch version bump.l (which was when I called it quits on ansible, I then relooked at terraform, and decided to go native)
I will caveat that our use case for AWS involves LOTS of creation and deletion, so I find it super helpful to manage my infrastructure in "stacks" that are created and deleted as a unit.. I dont need to worry about partial creations or deletions.. like ever... It basically never fails redoing known-working stuff... Only "first time" and usually because we follow least-privilege heavily
One is a configuration management and deployment tool.
The other one is cloud resource provisioning service.
They’re meant to work in tandem, not one to replace another.
The approach between Cloudformation/Terraform/Pulumi and Ansible are entirely different though.
The former are declarative, they define how the end state should look. Ansible is a task runner, you define a set of manual tasks it needs to execute to get to the end state.
I strongly advice against using Ansible for provisioning resources. It's idempotent by convention only. When I had to reluctantly use it for jobs it was extremely difficult to get a repeatable deterministic environment set up. Each execution lead to a different state and was just a nightmare to deal with.
Cloudformation/Terraform/Pulumi are much better in that regard as they generate a graph of what the end state should be, check the current state, generate an execution plan how to make the current state look like the target state.
Where Ansible is better than Cloudformation/Terraform/Pulumi is you have a bunch of servers already set up and you want to change the software installed/configuration on them. That's bit of an anti pattern these days changing config/provisioning at runtime. You can change that slightly and use Ansible with Packer to generate pre-baked images which works ok if you don't mind lots of yaml. This isn't to bad and works reasonably well and works to Ansible strengths all though these days most people don't prebake images with containerization. Also if you are only using Ansible for provisioning config on a host Nix achieves this much more elegantly / reliably.
Because ansible needs a trigger to start configuring the server.
The most general practice now is to use cloud-init (https://cloudinit.readthedocs.io/en/latest/)
For example, this is how u set it up on AWS - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-dat...
i think cloud-init is packaged by default on all providers OS images.
We already used ansible for other things, so it wasn't too hard to swap over to AWS modules... (Except they were inconsistent and poorly supported, we ultimately found out)
Someone at Hashicorp then convinced mgmt that terraform is almost a write-once system, and we could jump from AWS to Azure or GCP easily "just change the module!"... When actual engineers looked at it, after 3 days there was almost a mutiny and we rejected terraform mostly based on the fact someone lied to our managers to try and get us to adopt it... I know someone who is very happy with terraform nowadays, but that ship sailed for us.
Those were basically the only people in this space, so we started rewriting ansible to CloudFormation. Since we mostly use lambdas to trigger the creation of CF stacks, this really works well for us, since our lambdas exist for less than a second to execute, and then we can check in later to see if there's issues (which is less than 1 in 50,000? 100,000? in my experience... Except for core AWS outages which are irrespective of CF). Compared to our ansible (and limited terraform) setups which required us to run servers or ECS tasks to manage the deploy. We can currently auto-scale at lambda scale-up speed to create up to 30 stacks a second if demand surges (the stack might take 2-3 minutes to be ready, but it's now async). Under ansible/terraform we had to make more servers and worker nodes to watch the processes... And our deployment was .3/.4 stacks per minute per worker (and scaling up required us to make more workers before we could scale up for incoming requests)
If I was building today, I'd probably revisit terraform, but I think the cdk or CF are still what I'd recommend unless there's a need for more-than-AWS... E.g. multi-cloud deployments, or doing post-creation steps that can't be passed in by userdata / cloud-init.. in which case CF can't do the job alone and might not be the right tool.
This, 100%.
I then deployed such template through StackSets and bam, every account in specific OU automatically configures itself without any manual steps.
The problem is when those help tickets get responses like “try deleting everything by hand and see if it recreates without an error next time”. They've worked on CloudFormation over the last year or but everyone I've known who's switched to tools like Terraform did so after getting tired of unpredictable deployment times or hitting the many cases where CloudFormation gets itself into an irrecoverable state. I can count on no fingers the number of development teams who used CF and didn't ask for help recovering from an error state in CF which required out-of-band remediation.
I believe they've also gotten better at tracking new AWS features but there were multiple cases where using Terraform got you the ability to use a feature 6+ months ahead of CF.
> A portable Terraform + Kubernetes contraption is a lowest common denominator approach.
Terraform is much, much richer than CloudFormation so I'd compare it to CDK (with the usual aesthetic debate over declarative vs. procedural models) and it doesn't really make sense to call it LCD in the same way that you might use that to describe Kubernetes because it's not trying to build an abstraction which covers up the underlying platform details. Most of the Terraform I've written controls AWS but there's a significant value to also being able to use the same tool to control GCP, GitLab, Cloudflare, Docker, various enterprise tools, etc. with full access to native functionality.
However, you can use terraform just fine while still leveraging vendor specific services that actually offer added value, like DynamoDB or Lambda. Cloudformation however doesn't really offer that much added value (if any) over terraform, so using terraform isn't an LCD approach perse.
I found that especially puzzling because one of the reasons why we switched to Terraform was because it let us take advantage of new AWS features on average much faster than CloudFormation.
TF can be used as a managed service.
> Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away.
The same is true with TF, except 100000% better unless you're paying boatloads of money for higher tiered support.
> I only run workloads on AWS, so the CF syntax, specs and docs unlocks endless first party features.
CF syntax is an abomination. Lots of the bounds of CF are dogmatic and unhelpful.
> I have seen Terraform turn into a tire-fire of migrations from state files to Terraform enterprise to Atlantis that took an entire DevOps team to care for.
CF generally takes an entire DevOps team to care for, for any substantial project.
Sure, but I never seen that myself. If TF was used it was always own set up infrastructure at best.
> The same is true with TF, except 100000% better unless you're paying boatloads of money for higher tiered support.
Again, all places I worked had enterprise support and even rep assigned. I think I only used support for CF early on, I don't know if it was buggier back then or I just understood it better and didn't run into issues with it.
> CF syntax is an abomination. Lots of the bounds of CF are dogmatic and unhelpful.
I would agree with you if you were talking about JSON, but since they introduced YAML it is actually better than HCL. One great thing about YAML is that it can be easily generated programmatically without using templates. Things like Troposphere make it even better.
> CF generally takes an entire DevOps team to care for, for any substantial project.
Over nearly 10 years of my experience, I never seen that to be a case. I'm currently in a place that has an interesting approach: you're responsible for deployment of your app, so you can use whatever you want, but you're responsible for it.
So now I'm working with both. And IMO I see a lot of resources that are not being cleaned up (because there's no page like CF has, people often forget to deprovision stuff), also seeing bugs like for example TF needs to be run twice (I think last time I've seen it fail was that it was trying to set tags on a resource that wasn't fully created yet).
There is also situation that CF is just plain better. I mentioned in another comment how I managed to get datadog integration through a single CF file deployed through stackset (this basically ensured that any new account is properly configured). If I would end up using TF for this, I would likely have to write some kind of service that would listen for events from the control tower, whenever a new account was added to OU, then run terraform to configure resources on our side and make API call to DD to configure it to use them.
All I did was to write code that generated CF via troposphere and deploy it to stackset in a master account once.
What happens when you leave the project?
Right, your post is mostly "I like the thing that I've used, and I do not like the thing I haven't used". They're apples and different apples.
> Again, all places I worked had enterprise support and even rep assigned
So, again, you've worked at places that were deeply invested in CF workflows.
> but since they introduced YAML it is actually better than HCL. One great thing about YAML is that it can be easily generated programmatically without using templates.
Respectfully, this is the first-ever "yaml is good" post I think I've ever seen.
> Over nearly 10 years of my experience, I never seen that to be a case. I'm currently in a place that has an interesting approach: you're responsible for deployment of your app, so you can use whatever you want, but you're responsible for it.
I'd love to hear more about this.
> And IMO I see a lot of resources that are not being cleaned up (because there's no page like CF has, people often forget to deprovision stuff), also seeing bugs like for example TF needs to be run twice (I think last time I've seen it fail was that it was trying to set tags on a resource that wasn't fully created yet).
I guess we're just ignoring CF's rollback failures/delete failures/undeletable resources that require support tickets then?
> There is also situation that CF is just plain better. I mentioned in another comment how I managed to get datadog integration through a single CF file deployed through stackset (this basically ensured that any new account is properly configured). If I would end up using TF for this, I would likely have to write some kind of service that would listen for events from the control tower, whenever a new account was added to OU, then run terraform to configure resources on our side and make API call to DD to configure it to use them.
Again respectfully, yes, the person that both doesn't like and hasn't invested time into using Terraform at scale probably isn't going to find good solutions for complicated problems with it.
While this is true and AWS support is very responsive and useful, it doesn't mean they solve all the problems. Sometimes their help is: "I'll note that as a feature request, in the meantime you can implement this yourself using lambdas".
Cloudformation lags features so bad that you end up hacks or Lambda functions with custom resources. DynamoDB Global tables took 4/5 years to be available in Cloudformation.
I've also seen wrongly constructed Cloudformation deleting critical databases, hanging (often), times out, rollback hanging not succesful, so it's not always rainbows & sunshine there either. Also I don't like Terraform for its usage of excessive state files, handling state files with DynamoDB locks, having them on S3.
I won't deny the good features of it like being managed is a huge plus, but it's so slow and lagging behind, and it's YAML is so verbose an has stack size limits, it's always a workaround with Cloudformation. My company uses it by abstraction for internal PaaS and deployment automation and it takes a lot for trivial changes to complete.
So in short, neither is perfect, but for me Terraform is easier to use, easier to debug, and faster, and features don't lag nearly as CF. Those are good enough reasons to avoid Cloudformation for me. I also don't like CDK because it's too verbose and its still CF, and I would rather generate Terraform JSONs/HCLs myself if I need more logic either.
Terraform also helps when you need to configure multiple stacks i.e for a service you can have a module that reflects your organizational best practices, a Fargate service for runnning container, automatic Datadog dashboards, cloudwatch alarms connected to Opsgenie/Pagerduty etc.
In terms of getting new features fast CDK is strictly worse than CloudFormation.
Depending on how old the service is, the ownership of the CFN resource may be on the CFN service team (from back when they white-gloved everything) in which case there are massive development bottlenecks (there are campaigns to migrate these to service team ownership) or more often the resource is maintained by the service team itself, in which case the team may not be properly prioritizing CFN support. There can be a whole lot of crap to deal with for releasing a new CFN _resource_, though features within a resource are relatively easy.
On my last team, we did not consider an API feature delivered until it was available in CFN, and we typically turned it around within a couple of weeks of general API availability.
The timeouts in CF are ridiculous. Especially with app deployments. I can't remember which service it is, but something can wait up to 30min on a failed `apply` and then wait the same on a failed revert. Only then you can actually deploy the next version (as long as it wasn't part of the first deploy, then you get to wait until everything gets deleted as well).
(yes, in many cases you can override the timeouts, but let's be honest - who remembers them all on the first run or ever?)
The worst part is that there are no error messages. When it fails and I click "Details", it takes me to the stack page and shows 100% green. Support ticket seems slow to get a response too.
That aside, my overall experience has been positive!
DynamoDB global tables launched in November 2017, it isn't four years old yet.
CFN lag is an issue for sure but not quite that much of an issue...
If you want to use a new resource and feature, %90 likely it is an issue. There was even a public project to track them. https://github.com/cfntools/cloudformation-gaps/projects/1
You want to have an IAM role, but you can not tag it with Cloudformation. These minor frustrations quickly add up. And see what you need to do to add a custom resource: https://shouldroforion.medium.com/aws-cloudformation-doesnt-...
That being said the Vault provider is open source and it's quite easy to add it and roll your own.
Pretty sure that's the same problem with CloudFormation, albeit at a different scale: these service teams are providing APIs for this stuff already, so do they really want to spend time writing another layer of API on top? Ideally yes, but when you have deadlines you've gotta prioritize.
YAML programming is not something that makes people happy. K8s is even worse in this regard, YAML documents embedded into YAML documents.
https://github.com/awslabs/aws-cloudformation-templates/blob...
People are so hung up because we could do so much better with expressing only valid states and you do not need to deploy your infra to figure out that and S3 bucket cannot have a redirect_only flag set and also a website index document set at the same time.
The issue with CloudFormation is that it lags behind all other AWS services quite often. It seems to be maintained by another team. I realize that getting state management for complex inderdependent resources right requires time and diligence, BUT it's a factor in driving adoption.
- New EKS feature? Sorry, no knob in CF for MONTHS.
- New EBS gp3 volume type available? Sorry, our AWS::Another::Service resource does not accept gp3 values for months after feature release.
- A AWS API exposes information that you would like to use as an input to another resource or stack. SURPRISE, CloudFormation does not return that attribute to you, despite it being available. SORRY NOT FIXABLE.
- Try refactoring and moving resources in/out of stacks while preserving state? Welcome to a fresh hell.
- Quality of CloudFormation "modules" varies. AWS::ElasticSearch::Domain used to suck a whole lot, AWS::S3::Bucket as a core service was always very friendly.
- CloudFormation custom resources are a painful way of "extending" CF to suite your needs. So painful that I refuse to pay the cost of AWS not keeping their stuff up to date and well integrated.
These kinds of lag, this kind of incompleteness when it comes to making information from AWS APIs available have driven me to Terraform for things that are best done in Terraform and require flexibility and CloudFormation for things that work well with it.
At the end of the day CF is a closed-source flavour of Terraform + AWS provider. I would like have gone all in, but just doesn't work and costs hacks, time and flexibility.
That being said, if you have no idea how to work with TF, tell the devs to use CF.
The experience of AWS support is probably also very different when it comes to feature requests. An org that spends half a billion with AWS will get their requests implemented ASAP whereas small fish have to swim with the flow and hope it works for them.
It is also very possible to get into very bad situations if your settings drift and you attempt to reconcile those changes.
CloudFormation is great because of its transactionality, so it lends itself nicely to deploying multiple services which are versioned together. You either succeed fully, or all services will be rolled back.
This way you can deploy your whole infra with Terraform, and then deploy to your i.e. ECS cluster using CloudFormation. Works great in practice.
I've spent hours just waiting for CF to fail deleting EKS or RDS related resources then I end up getting billed for $30+ a month sometimes because I forgot to manually delete a NAT gateway.
When it works, which is a big caveat: we had far more cases where it failed in a way which required manual remediation and the gaps in validation meant that you'd be in a “apply / error / rollback” loop requiring 20+ minutes before you could try again. Terraform was always considerably faster but it was especially the orders of magnitude improvement in retry time which convinced most of us to switch.
The CloudFormation team has been working on this so it's possible that experience has improved but the scar tissue will take time to fade.
This is especially true for things like CloudFormation where the user is likely on a deadline or in a stressful situation trying to fix a problem or hit a deployment window. Forcing someone to wait an unpredictable amount of time for no reason ramps up the stress level massively.
The issue could be at least partially resolved by using automation (like atlantis for example) to apply your plans.
Ok, so apparently I need to write a custom Cloudformation resource to execute a lambda function that will run the migrations prior to deploying the new version of the lambda. Kind of neat that you can do that.
Except I messed up the output of the custom resource lambda and Cloudformation completely locked my deployment up for 3 hours. 3 hours. I couldn't do _anything_ - rollback, update, whatever.
Cloudformation via a CDK is interesting, and I don't hate it, but oh boy if it gets into a weird state it can completely kill your iteration loop. And the docs say something along the lines of "if it's stuck for too long contact support". No thanks.
I find it easiest to run migrations when an application is first starting up (with an appropriate transaction lock so other instances won't cause the migration to run more than once), this way you don't have to do a lot of devops magic for it to work.
I guess I could do the migration check in the lambda startup rather than in the request path, but it’s still not ideal especially when handling rollbacks.
I'd pass in DB settings as parameters to the SAM file, rendering them as env. variables for the lambdas. with DB access managed by IAM roles (so you don't need sensitive info in the lambda env).
And depending on size/scope of project/team, might have one CF for the DB infra, and another for the app lambdas.
Using IaC services means I can have both dev and prod infra, so I can test DB migrations.
And I can do this as a solo dev on a side project where I'm also responsible for front-end and the other back-end stuff plus keeping clients happy on evenings/weekends.
The upside is that I got really well acquainted with how ECS worked.
Our commitment to CloudFormation was doubled down on that we could do containers, Lambda, and 95% of any other AWS Services....
However, in hidsight using SAM and the ECS CLI probably would have resulted in a more predictable CI/CD process as we weren't fighting deploy semantics through CloudFormation abstraction.
CFn will allow you to specify up to 5 alarms to monitor during stack deployment as well as a duration to monitor them for up to 3 hours.
During deployment, the stack does not transition into the COMPLETE state until the monitor duration has completed without the alarms being triggered. If one of the alarms is triggered, the stack immediately rolls back.
Since failed ECS tasks create CW Events, you can set up alarms on them and use them as the rollback monitor trigger.
CDK is superior to terraform for a glaring reason: it's a first class citizen in AWS' eyes and terraform is not.
That's just not how reality works. First class citizens can suck, because guess what, they're designed by mortals
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
See: https://www.pulumi.com/docs/intro/concepts/function-serializ...
Seeing the specific examples they use it for (AWS infra glue) makes me think that there is room for infrastructure related lambdas to be defined right in cfn or infra code, with very low ceremony, even if you wouldn't want to deploy "applications" like that.
So deploying to an AWS etc. Lambda seems conceptually similar to deploying from static-site-generator to a static web-site source-code, which is then what the browser will execute. Or is it?
I think the state-of-the-art is Kubernetes + jsonnet, though you still need to suffer through the learning curve and you'll still need TF and company to provision some of your cloud resources.
https://www.pulumi.com/blog/pulumiup-native-providers/
The engine is entirely separate and, now that they have a foothold with it, they are building out providers that are built around the Pulumi execution model first-class, rather than an adapter layer.
One notable core difference in the Pulumi model is that it doesn't have "plan" in the same sense that Terraform does, where the set of actions is fixed and just executed once you accept it, returning an error if that turns out not to be possible.
"pulumi preview" just runs the program in a dry-run mode, so there's no guarantee that the subsequent apply will match unless you've been careful to write a totally deterministic program that always behaves the same with placeholder values as it does for concrete values.
That's a typical tradeoff for using a DSL vs. a general-purpose language though, and one that seems to pay off for a lot of teams.
But there are indeed repositories in their GitHub org which started as forks of Terraform providers, as I said. Pretty good strategy to start from all of the generic glue code somebody else already wrote and then adapt it to a new engine, because that's the undifferentiated heavy lifting that needs to be done regardless of the engine design.
I like that HCL is declarative and deliberately restricted in what’s possible, it means I never have to think about some dev writing “clever” code or over abstracting things, or writing bugs that spin up to many machines.
Once you’re working with HCL templated via Gomplate or what have you, and a pipeline which processes the code base and then spits out the resultant HCL as an artifact to be used by a subsequent step, you’ll pine for the impenetrable JS or Python code. At least that is code which has a reasonable development workflow associated with it, not a monstrosity held together with duct tape and vulgar hacks.
Pulumi does work very well in the Azure space as Azure resources are already well abstracted.
I get that CloudFormation may be a bit better than Terraform in this way or that way. But I have no personal interest in learning it. All that investment of time and effort and when I need to do a project on a different vendor I have nothing, no skills. What's the point?
To be clear: I understand why companies and even specific teams or projects may choose to go all in on vendor-lock-in. It's often the most direct line to the business value they want and as people point out, it's rare to never that most cloud native applications will shift cloud vendors, and even if they do most of a Terraform setup will have to be reworked.
So I get why companies and managers want that. I just don't understand why software engineers in those teams are so happy to go along.
I also don’t understand those organisations that use every single AWS managed solution (RDS, Redis, Elasticsearch, EKS) but then they feel the need to manage their infrastructure state themselves with Terraform instead of choosing the managed solution for this.