This blog feels like a poor ad, I was hoping for technical details but seems like this tool just swept in and 'saved the day'. I have no idea how
* Provisioning time dropped from hours to minutes.
* Debugging speed improved because we could fix it in real time.
happened.
Seems like the problem of a long feedback loop would have been solved by pull request preview environments and or enabling developers to have their own deployed instances for testing etc.
Ops type (DevOps/SRE/Sysadmin/whatever you want to call me) here, so I was really interested and this blog left me with more questions than answers?
What is SI? Homegrown GUI Terraform? That part is not clear in article. It looks like homegrown GUI Terraform with module so that's what I'm going with. Cool, glad you got that working, sounds like a big project and you were able to pull it off.
However, this part confused me, "Our engineers were investing a lot of time in what felt like “IaC limbo,” making a change in a Terraform file, waiting for review, waiting for CI/CD to run, and only then finding out if it worked. A simple tweak to a networking rule could take hours to validate."
What in tarnation are you doing? Do you have massive terraform file repo so apply takes forever since the plan is running forever? Talk to me Goose, what is going on that Terraform changes take hours to run? Our worst folder takes about 10 minutes to plan because it's massive "everything for this specific project". We also let people run tofu plan/apply from their laptops in Dev so feedback is instant.
We do have folders that have dependency on others folder, for example, we can't setup Azure Kubernetes without network being in place but we just left dependson yaml that our CI/CD pipelines work off when doing full rollout which is not their normal mode of operation (it's for DR only). We also assume that people have not been ClickOps either or if they have, they take responsibility for letting IaC resolve it.
Writing your own API calls to Cloud Provider is not something I would wish upon anyone. I did it for Prometheus HTTP Service Discovery system and just getting data was difficult, I can't imagine Create/Update/Delete.
3 comments
[ 3.7 ms ] story [ 24.7 ms ] thread* Provisioning time dropped from hours to minutes. * Debugging speed improved because we could fix it in real time.
happened.
Seems like the problem of a long feedback loop would have been solved by pull request preview environments and or enabling developers to have their own deployed instances for testing etc.
What is SI? Homegrown GUI Terraform? That part is not clear in article. It looks like homegrown GUI Terraform with module so that's what I'm going with. Cool, glad you got that working, sounds like a big project and you were able to pull it off.
However, this part confused me, "Our engineers were investing a lot of time in what felt like “IaC limbo,” making a change in a Terraform file, waiting for review, waiting for CI/CD to run, and only then finding out if it worked. A simple tweak to a networking rule could take hours to validate."
What in tarnation are you doing? Do you have massive terraform file repo so apply takes forever since the plan is running forever? Talk to me Goose, what is going on that Terraform changes take hours to run? Our worst folder takes about 10 minutes to plan because it's massive "everything for this specific project". We also let people run tofu plan/apply from their laptops in Dev so feedback is instant.
We do have folders that have dependency on others folder, for example, we can't setup Azure Kubernetes without network being in place but we just left dependson yaml that our CI/CD pipelines work off when doing full rollout which is not their normal mode of operation (it's for DR only). We also assume that people have not been ClickOps either or if they have, they take responsibility for letting IaC resolve it.
Writing your own API calls to Cloud Provider is not something I would wish upon anyone. I did it for Prometheus HTTP Service Discovery system and just getting data was difficult, I can't imagine Create/Update/Delete.