Putting yourself in a situation where this could happen is kinda insane, right? Could be something I'm missing.
I can't think of any specific example where I would let any agent touch a production environment, the least of which, data. AI aside, doing any major changes makes sense to do in a dev/staging/preview environment first.
Not really sure what the lesson would be here. Don't punch yourself in the face repeatedly?
But that's the "promise" of AI (that management believes), isn't it? That it can replace an engineer because it's as good or better -- so why wouldn't you allow it to touch your production database?
(I agree with you, just pointing out the difference between what's being sold and reality.)
You wiped your production database. You actively ignored the warnings of your tooling and your backup strategy was bad. Incompetence as content is surging in the last few weeks.
Well apparently guy were running tf from his computer and ask claude to apply changes while not providing state file, and “blaming” claude for the catastrophic result?
Hah, perfect... some variation of "From now on [your AI tool of choice] is banned. Just write code like a normal human f*king being, please." has probably already been used in the real world recently.
Despite multiple comments blaming the AI agent, I think it's the backups that are the problem here, right? With backups, almost any destructive action can be rolled back, whether it's from a dumb robot, a mistaken junior, or a sleep-deprived senior. Without, you're sort of running the clock waiting for disaster.
I don’t use Terraform much anymore because don’t need it but that’s not how you use it.
Always forward evolve infra. Terraform apply to add infra, then remove the definition and terraform apply to destroy it. There’s no use in running terraform destroy directly on a routine basis.
Also, I assume you defined RDS snapshots also in the same state? This is clearly erroneous. It means a malformed apply human or agent results in snapshot deletion.
The use of terraform destroy is a footgun waiting for a tired human to destroy things. The lesson has nothing to do with agent.
One of Terraform's most powerful features that it will tell exactly which resources change before it makes the changes. The hard part is writing Terraform, not reviewing and running one command. In my workflows I am the one who runs "terraform apply", NOT the agent.
> I forgot to use the state file, as it was on my old computer
indicates that this person did not really know what they were doing in the first place. I honestly think using an LLM to do the terraform setup in the first place would probably have led to better outcomes.
I blame not only the engineer who ran the command, Claude which made the mistake, but also software engineers as a group (because Terraform is way too dangerous a tool to be used by engineers and not dedicated SREs, yet we have somehow made this the default. I'm happy to be convinced otherwise, but I've seen enough carnage when "senior" engineers fuck up terraform that it'll be difficult), and also I blame cloud platforms like AWS that are overly complex and led to the Claude confusion.
There was a project at Ansible that aimed to address this kinda thing when I worked there. The idea was to write policy as code definitions that would prevent users (or AI) from running certain types of automation. I don’t know where that project ended up but reading about this makes me think that they were on to something.
83 comments
[ 3.2 ms ] story [ 87.7 ms ] threadI can't think of any specific example where I would let any agent touch a production environment, the least of which, data. AI aside, doing any major changes makes sense to do in a dev/staging/preview environment first.
Not really sure what the lesson would be here. Don't punch yourself in the face repeatedly?
Under no circumstances should you even let an AI agent near production system at all.
Absolutely irresponsible.
Always forward evolve infra. Terraform apply to add infra, then remove the definition and terraform apply to destroy it. There’s no use in running terraform destroy directly on a routine basis.
Also, I assume you defined RDS snapshots also in the same state? This is clearly erroneous. It means a malformed apply human or agent results in snapshot deletion.
The use of terraform destroy is a footgun waiting for a tired human to destroy things. The lesson has nothing to do with agent.
I'd say skill issue.
> If you found this post helpful, follow me for more content like this.
> I publish a weekly newsletter where I share practical insights on data and AI.
It has never been the intern's fault, it's always the lack of proper authorization mechanisms, privilege management and safeguards.
Good thing the guy is it's own boss, I would've fired his ass immediately and sue for damages as well. This is 100% neglectful behavior.