I'll say straight away, IaC for things like AWS in my company at least is invaluable.. the thought of even trying to deploy services with user policies etc, let alone the mess people would make of that is a disaster.
Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.
With the same logic you can say that contraceptives are not the answer because they aren't 100% effective.
Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.
Imho, I think we've gone a bit overboard with Terraform at my company. Sure, it'll automate a bunch of stuff, but only a handful of people will really "get" it.
I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case.
I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.
I often see people making what could be very simple terraform setups into incompehensible monolithic monsters as well.
I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
The opposite can be true as well. I’ve seen TF environments that were designed with so many layers of abstractions that unraveling a single piece of infra was an incredibly long endeavor.
Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
The network effects benefits of a large population using a common domain specific language and abstraction are easily overlooked when one on the inside of that echo chamber dealing with the warts of that context frame. I’m also not overly fond of the current state of IaC but we have a good enough foundation upon which I have a rough line of sight on a future where we develop better representations going forward, and that alone is enough for me to avoid wholesale tossing it out or halting/slowing its use.
The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these.
> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.
That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs.
You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production.
Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.
This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race).
To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.
If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.
It should have never been called infra-as-code, but infra-as-data.
This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code.
Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases.
Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
It doesnt absolve the whole article but a delete command for your entire infrastructure is obviously a major footgun that doesnt exist with alternatives (to use one of your own lenses).
Or if you just use the terraform cli properly and actually glance at the report it gives you before the “hey guy are you really, really sure about this?” Safety prompt.
Eh, Azure has a big delete button for any Subscription. Seen it happen and take down a production system.
Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
The "as code" means the quality and stability of the implementation varies wildly. If you use a declarative style with proper SDLC in place, then it's fantastic.
On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
> If you do 'DROP TABLE' in production, nothing can save you.
Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
That’s not infrastructure as code means. It means engineers wouldn’t have ability to run terraform on laptop at all. They have to make a PR, get it reviewed and the code lands in some prod level branch where CI would run terraform for them.
And if things got messed up, revert the commit and we are back (mostly - May need backup restores).
It means your infrastructure is a commit that you can see history and forward/reverse.
Git represents your infra. If it’s not in the repo, it’s not there.
We also went from - click around in the gui -> automate everything -> click around in the gui. Zero regrets.
We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk.
Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too.
To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost.
When you have 10 boxes to manage doing it manually is within the realm of 1 person doing it and maybe even correctly! When you have 500... You will want consistency and repeatability. To this day I am still 'fixing' things someone did in the 10 machine area. 'oh yeah that machine is set up this way and the others are that way'. But I can not get rid of those 10 boxes because they became key infrastructure.
we use elastic beanstalk which is functionally not that different than elastic container service/Fargate. It’s basically Heroku when all is said and done.
Everything sits behind application load balancer and API gateway, all basic stuff
We have about 100 boxes, but basically only 3 “environments” for our distributed monolith. Every box in a distinct environment is provisioned and configured in exactly the same way.
need to change some config? It’s at most a few extra clicks to do it in multiple environments. Then EB handles the rolling deployments and health check failover stuff. It’s very hands off, with another small benefit of not needing to bother with fudging around with terraform state, wasting compute on building containers and paying for registry storage.
I don't know much about Elastic Beanstalk but bare EC2 instances connected to the Internet should fill anyone with dread. Securing the O/S and the network is not trivial and AWS will gladly let you do the wrong thing out of the box.
IaC doesn't magically solve any of those problems but if you make a mistake once you can fix it once instead of fixing it N times.
Besides the point, but Medium.com is not the answer if you want to publish text on the internet.
As for the article he's not proposing anything better.
Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.
Decent GUIs typically have sanity checks and validation built in. Yaml otoh has an extremely relaxed structure, which to me sounds dangerously easy to mess up. Terraform isn’t my domain, but do senior yaml engineers these days have proper guardrails against typos, wrong indentation, type errors etc?
Works fine if you have a minimal scope, but try that with multiple teams of engineers who need to each deploy their piece of org level infra and services, and it will be a mess
I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc.
But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important.
I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.
I disagree with a lot of this, but here are the bit i really disagree with:
> The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there.
This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS.
Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood.
> It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read.
Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.
Letting teams self-serve XaaS is how you get 50 dramatically mis-sized and ill-configured DB instances in prod. This team thought they’d have 1000 QPS, and have an instance that’s 50x as large as necessary. This team doesn’t know what IOPS are but figured they’d better have a lot of them, so they’re paying thousands per month for an io2 drive when the entire dataset fits comfortably into memory. This team has no idea (fair – there are a million knobs) how to configure Postgres, and the defaults suck, so they’re getting 1/4 the performance that they’re paying for.
IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.
IAC is one technology I will defend to the hilt. IAC is difficult because the underlying problem it is trying to solve is hard. Different people over time are making updates to a mission-critical, stateful, system that need to be coordinated with each other.
IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change.
If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.
Setting up solutions in AWS is a most painful experience.
Even AWS consultants do not find it easy and trivial.
However, IaS being a complicated hornets' nest is more
direct evidence of vast complexity in architecture.
The best way to address it is to decrease the complexity
of the system. It will have great rewards across the board.
Most systems composed these days, have far more moving parts
than what is needed.
Simplify and shrink.
Some projects really do need it, no doubt about that, but I
predict that somewhere around 80% at least coud make do with
far less, and the project will benefit from it to to bottom.
tbh this is the only sound comment in this whole thread.
IaC is literally a million times better than the alternative.
sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
Maybe it is that IaC is usually imposed on developers, injecting a forced learning curve. In this respect, IaC is a lot like JavaScript, and I suspect that, much like JavaScript, people are just mad about being forced to learn it and haven’t realized all the “good parts” yet.
I read the addendum. I think it was a good, understandable rant.
When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself.
If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code.
I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease.
But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.
Terraform isn’t Ansible, and vice-versa. I’ve seen both being shoehorned to do the job of the other, and it isn’t pretty.
Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications.
Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
Sure, there are many tools in this area, and they can be used in many ways.
We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz.
As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.
Not sure about the vice versa part. Ansible is pretty much a superset of Terraform. I've used in multiple places to manage vm:s and other cloud services and generally been happy with the setup.
Regardless of what the author himself is saying, this is a clear example of someone that did not actually get passed intermediate level in knowledge and expertise.
IaC is undeniably a good pattern for dealing with complex systems. The bigger problem is that it's not a solution for delineating responsibilities between software engineers and infrastructure engineers. All the specifics about an environment that need to make their way from software->infra (and in some cases the other way) still need to be worked out, and that's often a bigger problem than getting code to a box that always has access to a database (oversimplification).
I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications.
72 comments
[ 3.5 ms ] story [ 134 ms ] threadTerraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.
Good automation would be where you need less manpower afterwards..
Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.
Anyway... back to scale, consistency, and speed - IAC.
I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case.
I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.
I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.
That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.
You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.
This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.
If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.
It should have never been called infra-as-code, but infra-as-data.
See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/
Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative.
Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.
You not paying attention is not terraforms fault.
Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
* it throws a.big warning text
* it lists out all the resources you're deleting
* you have to type out the name of the subscription
* click another button
* and then another yes I'm sure button with more warning text
At the end of the day they have to provide a button.
IaC is much better than what it replaced, even if there are still many sharp corners.
On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
And if things got messed up, revert the commit and we are back (mostly - May need backup restores).
It means your infrastructure is a commit that you can see history and forward/reverse.
Git represents your infra. If it’s not in the repo, it’s not there.
We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk.
Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too.
To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost.
YMMV of course.
Everything sits behind application load balancer and API gateway, all basic stuff
We have about 100 boxes, but basically only 3 “environments” for our distributed monolith. Every box in a distinct environment is provisioned and configured in exactly the same way.
need to change some config? It’s at most a few extra clicks to do it in multiple environments. Then EB handles the rolling deployments and health check failover stuff. It’s very hands off, with another small benefit of not needing to bother with fudging around with terraform state, wasting compute on building containers and paying for registry storage.
IaC doesn't magically solve any of those problems but if you make a mistake once you can fix it once instead of fixing it N times.
As for the article he's not proposing anything better.
Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.
Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.
Give me Fly.io or something similar any day. I'll pay the premium gratefully.
GL clicking your way through that buddy.
> Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available.
This is true for all code and we have tried and tested methods/principles for dealing with that.
> Someone’s user account got built in, and they don’t work here any more.
No comment needed.
> He was in the Prod directory — and starting nuking production.
Say it with me: Elevate Permissions
I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc.
But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important.
I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.
> The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there.
This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS.
Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood.
> It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read.
Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.
IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.
IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change.
If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.
However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture.
The best way to address it is to decrease the complexity of the system. It will have great rewards across the board.
Most systems composed these days, have far more moving parts than what is needed.
Simplify and shrink.
Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.
self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook
prevents mistakes - maybe. but at least the mistakes are documented and visible.
prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close
IaC is literally a million times better than the alternative.
sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself.
If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code.
I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease.
But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.
Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications.
Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz.
As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.
I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications.
1. https://noop.dev/blog/build-apps-not-infrastructure/
These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.