Ansible is just extremely easy to begin with, and comfortable to use since it is an agentless solution using SSH. As for Puppet, well, it could largely depends on your team. Is it a devops one or a strictly dev one? Puppet seems to be the perfect balance for us (devops mostly, but devs can touch it with confident too)
I enjoy using mage (https://github.com/magefile/mage). I like having a full language at my disposal for configuring things rather than yaml or json or whatever else.
(To me things like puppet or ansible seem like thin layers over shell and ssh, whereas both Dhall and Cue seem to innovate in ways that are more, uh, je ne sais quoi ;-) YMMV)
This is the approach we take. We don't track states or do continuous config management either as we're all in on cattle > pets (and we don't typically have the time to maintain terraforms properly enough to do anything but cut new environments). Something gets sick? Shoot it and stand up another one.
I think this is the ideal scenario for Ansible— one-time configuration of throwaway environments, basically as a more hygenic and structured alternative to shell scripts.
My experience trying to manage longer lived systems like robot computers over time with Ansible has been that it quickly becomes a nightmare as your playbook grows cruft to try to account for the various states the target may be coming from.
Could you say more about why ansible is better than shell scripts for one-time configuration? In my mind, ansible's big advantage over shell scripts is that it has good support for making incremental changes to the configuration of existing resources. In a situation like packer, where the configuration script only gets run once, I prefer the conciseness of a shell script.
That's a very good point : I also found that the core feature of configuration management - idempotency - actually becomes mostly useless in this case, as ansible applies a playbook only once.
I still use it as it allows more portability across OS releases and families (as in easier migration), but it also increases the complexity when creating a new task/role/playbook.
In that sense, Dockerfiles with shell-based RUN commands are much easier to manage.
Another advantage of config management over shell might be a better integration with the underlying cloud provider. For instance Ansible supports AWS SSM parameter store, which allows me to use dynamic definitions of some configuration data (RDS database endpoints, for instance) or secrets (no need for Ansible vault)
I see the incremental piece as a dev-time bonus rather than something to try to leverage much in production— it lets you iterate more quickly against an already-there target, but that target is still basically pristine in that any accumulated state is well understood. But that's very much not the case if you're trying to do an Ansible-driven incremental change against a machine that was deployed weeks or months earlier.
Even in the run-once case, though, I think there's a benefit to Ansible's role-based approach to modularization. And again for the dev scenario, it's much easier to run only portions of a playbook than it is to run portions of a shell script.
And finally, the diagnostics and overall failure story are obviously way better for Ansible, too.
Now, all this said, I do still go back and forth. For example, literally right now in another window I'm working a small wrapper that prepares clean environments to build patched Ubuntu kernels in— and it's all just debootstrap, systemd-nspawn, and a bunch of shell script glue.
I still prefer the Open Source edition of https://puppet.com/ to manage larger, diverse environments - which may include not just servers, but workstations, network appliances and so on. It's well established with lots of quite portable modules. But it can also be a bit on the slower side and comes with a steeper learning curve then some of the others.
https://www.ansible.com/ is surely a good solution for Bootstraping Linux cloud machines and can be quite flexible. I personally feel like its usage of YAML manifests instead of a domain-specific language can make complex playbooks harder to read and to maintain.
If all you do is to deploy containers on a managed Kubernetes or a similar platform, you might get away with some solution to YAML templating (jsonnet et al) and some shell glue.
I am keeping an eye on https://github.com/purpleidea/mgmt which is a newer contender which many interesting features but lacks more complex examples.
Others like saltstack and chef still see some usage as far as I know, but I've got no personal experience with them.
I used to use Puppet back when they were ruby based. I dropped them once they switched to Java, not interested in pushing Java onto every host when it's not in our stack.
It's still good in Enterprise land where taking the time to work out the declarative style and dependency chains is worth it (and you have the people to put on it and the CAB process to review infrastructure changes). For a small to mid sized company I find it gets in the way of iterating fast. I spent waaaay too much time there either fighting the tooling or having to work out dependency chains. Redhat and I-think-AWS-but-I-might-be-thinking-of-Chef also have tooling in this space.
I'll take Chef or Ansible's imperative approach in the environments I work in any day (Mostly ansible playbooks for baking hosts only, I've never been entirely comfortable with having one Ansible Tower/Chef Server/Puppetmaster/etc be authoritative over everything, too large a failure pattern if security controls fail). But again, I'm working in many younger small environments and not large mature ones.
Most of this is also irrelevant for us as we're all in on Docker/ECS for anything new. Config management plays a limited role there over having your tasks/services checked into the individual repos.
Just for reference, the clients are all still ruby based. It's only the web servers for the puppet masters ( the parsing code is still jRuby ) and puppetdb that are written in clojure that runs on the JVM.
Ansible amazing for configuration management, much better than Puppet. Storing the config in YAML makes it super easy to read and maintain, also much better than Puppets method.
As you mention, puppet has a steep learning curve, whereas Ansible has a very shallow one. It’s easy to get running in a few minutes!
We use both Puppet and Ansible at work, and its constant complaints and delays with Puppet whereas Ansible is little complaints and no delays.
The master is part of the bottleneck, but a lot of the complaints are trying to get it to do what it says. But a big benefit of puppet is the master feature, so if that's taken out, why puppet?
I use Ansible, mostly because it works pretty well for deployments (on traditional, non-dockerized applications), and then I can just gradually put more configuration under management.
So it's a very good tool to gradually get a legacy system under configuration management and thus source control.
I'm curious why people use configuration management software in 2020. All of that seems like the old way of approaching problems to me.
What I prefer to do is use Terraform to create immutable infrastructure from code. CoreOS and most Linux variants can be configured at boot time (cloud-config, Ignition, etc) to start and run a certain workload. Ideally, all of your workloads would be containerised, so there's no need for configuration drift, or for any management software to be running on the box. If you need to update something, create the next version of your immutable machine and replace the existing ones.
But... How do you configure the hosts where your containers are running on? How do you configure your storage (NAS/SAN)? How do you configure your routers and switches? ...
The original question didn't have much context, and I guess my answer assumed someone would be using a cloud provider as opposed to anything on premise.
Are Ansible/Puppet/Chef any good for managing the hardware you mentioned?
The question asks what I would consider to be the right approach for 2020, and also what my team is doing. This is the design pattern I've been following for 5 years, but obviously your mileage may vary, it won't work for everyone, etc.
I mostly don't for new stuff (all in on Docker/ECS), however we have a lot of old stuff and things in the process of being migrated where it makes sense. There's also always the odd bird thing you use that needs to run on a regular host.
Example: How do you do "immutable" management of Mac OS machines? Taking what's typically described as such there, you've just turned a 30s deploy of software into a multi-hour "lets reimage the entire machine"?
(although that's of course not strictly "old stuff")
Were Macs in scope of the original post? I assumed it was server side stuff, rather than office hardware. For that, though, I'd use Jamf (Pro) or some other MDM option.
Even if you exclude "office hardware" from configuration management, our Mac OS build and test farm is "servers" I'd say. Not everyone running servers is doing so to run an online service on a platform of their choice.
I would have to suss out dependency chains on stuff that others built.
Rather than doing that I take a light touch approach with ansible that will suffice until we can dockerize (which would require the same work, but then it’s a dev project vs now when it’s just a devops thing)
"Immutable infrastructure" what a laugh. In a large deployment, configuration somewhere is always changing - preferably without restarting tasks because they're constantly loaded. We have (most) configuration under source control, and during the west-coast work day it is practically impossible to commit a change without hitting conflicts and having to rebase. Then there are machines not running production workloads, such as development machines or employees' laptops, which still need to have their configuration managed. Are you going to "immutable infrastructure" everyone's laptops?
(Context: my team manages dozens of clusters, each with a score of services across thousands of physical hosts. Every minute of every day, multiple things are being scaled up or down, tuned, rearranged to deal with hardware faults or upgrades, new features rolled out, etc. Far from being immutable, this infrastructure is remarkably fluid because that's the only way to run things at such scale.)
Beware of Chesterton's Fence. Just because you haven't learned the reasons for something doesn't mean it's wrong, and the new shiny often re-introduces problems that were already solved (along with some of its own) because of that attitude.
The hosts are managed via chef, the jobs/tasks running on those hosts by something roughly equivalent to k8s.
As for the conflicts, I have to say I loathe the way the more dynamic part of configuration works. It might be the most ill conceived and poorly implemented system I've seen in 30+ years of working in the industry. Granted, it does basically work, but at the cost of wasting thousands of engineers' time every day. The conflicts occur because (a) it abuses source control as its underlying mechanism and (b) it generates the actual configs (what gets shipped to the affected machines) from the user-provided versions in a non-deterministic way which causes spurious differences. All of its goals - auditability, validation, canaries, caching, etc. - could be achieved without such aggravation if the initial design hadn't been so mind-bogglingly stupid.
But I digress. Sorry not sorry. ;) To answer your question, my personal solution is to take advantage of the fact that I'm on the US east coast and commit most of my changes before everybody else gets active.
Sometimes you have to work with what you're given in a brownfield env and a config managment tool is useful in that case, but it's possible that you are working with a less than ideal architecture with less than ideal time/money to make changes.
State is always the enemy in technology.
I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
> can't even imagine managing hundreds of servers whose state is unpredictable at any moment
Be careful not to conflate immutability with predictability. The state of these servers is predictable. All of the information necessary to reconstruct them is on a single continuous timeline in source control. But that doesn't mean they're immutable because the head of that timeline is moving very rapidly.
> can't be terminated and replaced with a fresh instance for fear of losing something.
No, there's (almost) no danger of losing any data because everything's erasure-coded at a level of redundancy that most people find surprising until they learn the reasons (e.g. large-scale electrical outages). But there's definitely a danger of losing availability. You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming. Rolling changes are an absolute requirement. Some take minutes. Some take hours. Some take days. Many of these services have run continuously for years, barely resembling the code or config they had when they first started, and their users wouldn't have it any other way. It might be hard to imagine, but it's an every-day reality for my team.
> Be careful not to conflate immutability with predictability.
I don't trust predictability. Drift is always a nightmare. Nothing is ever as predictable as you would like it to be.
>You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming.
Except that state and its manipulation is usually the primary value in technology.
> I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
Yes, that sounds awful. That's why we have backups and, if necessary, redundancy and high availability.
Are you sure you two are talking about the same thing?
My understanding of immutable infrastructure is the same as immutable data structures: once you create something, you don't mess with it. If you need a different something, you create a new one and destroy the old one.
That doesn't mean that the whole picture isn't changing all the time. Indeed, I think immutability makes systems overall more fluid, because it's easier to reason about changes. Mutability adds a lot of complexity, and when mutable things interact, the number of corner cases grows very quickly. In those circumstances, people can easily learn to fear change, which drastically reduces fluidity.
Yup. We do this. When our servers need a change, we change the AMI for example, and then re-deployment just replaces everything. Most servers survive a day, or a few hours.
Makes sense to me. I was talking with a group of CTOs a couple years back. One of mentioned that they had things set up that any machine more than 30 days old was automatically murdered, and others chimed in with similar takes.
It seemed like a fine idea to me. The best way to be sure that everything can be rebuilt is to regularly rebuild everything. It also solves some security problems, simplifies maintenance, and allows people to be braver around updates.
Probably the most insightful comment in this entire thread. Thank you. In many cases, an "image" is just a snapshot of what configuration management (perhaps not called such but still) gives you. As with compiled programming languages, though, doing it at build time makes future change significantly slower and more expensive. Supposedly this is for the sake of consistency and reproducibility, but since those are achievable by other means it's a false tradeoff. In real deployments, this just turns configuration drift into container sprawl.
So, once you create a multi-thousand-node storage cluster, if you need to change some configuration, replace the whole thing? Even if you replace onto the same machines - because that's where the data is - that's an unacceptable loss of availability. Maybe that works for a "stateless" service, but for those who actually solve persistence instead of passing the buck it just won't fly.
Could you say more about why your particular service can't tolerate rolling replacement of nodes? You're going to have to rebuild nodes eventually, so it seems to me that you might as well get good at it.
And just to be clear, I'm very willing to believe that your particular legacy setup isn't a good match for cattle-not-pets practices. But I think that's different than saying it's impossible for anybody to bring an immutable approach to things like storage.
The person you're replying to didn't say "replace every node," they said "replace the whole thing."
To give a really silly example, adding a node to a cluster is a configuration change. It wouldn't make sense to destroy the cluster and recreate it to add a new node. There are lots of examples like this where if you took the idea of immutable infrastructure to the extreme it would result in really large wastes of effort.
Could you please point me at prominent advocates of immutable infrastructure who propose destroying whole clusters to add a node? Because from what I've seen, that's a total misunderstanding.
As I said, it's a silly example just to highlight an extreme. In between there are more fluid examples. I don't think it's that ridiculous to propose destroying and recreating the cluster in its entirety when you're deploying a new node image. However as you say I'm not sure anyone would advocate that except in specific circumstances.
On the other hand, while my suggestion of doing it to add a node sounds ridiculous I'm sure there are circumstances in which it's not only understandable but necessary, due to some aspect of the system.
I'm saying it's not even an extreme, in that I don't believe what people are calling "immutable infrastructure" includes that.
If your biggest objection to an idea is that you can make up a silly thing that sounds like it might be related, I'm not understanding why we need to have this discussion. I'd like to focus on real issues, thanks.
I'm not objecting categorically to anything. I think that immutable infrastructure is a spectrum, and depending on your needs you may have just about everything immutably configured, or almost nothing. I just don't think it's so black and white as "you should always use immutable infrastructure."
I also think it's a cool idea to destroy the entire cluster just to add a node, and it sounds ridiculous but also like there's some circumstances where it makes perfect sense.
Again, do you have a citation for the notion that it's a spectrum? The original post that coined the term doesn't talk about it that way, and neither do the other resources I found in a quick search. As I see it, it's binary: when you need to change something on a server, you either tinker with the existing server or you replace the server with a fresh-built one that conforms to the new desire.
Wow, look at those goalposts go! If you make enough exceptions to allow incremental change, then "immutable" gets watered down to total meaninglessness. That's not an interesting conversation. This conversation is about configuration management, which is still needed in a "weakly immutable" world.
Again, could you please point me at notable advocates of immutable infrastructure proposing the approach you take such exception to? And note that I'm not proposing any exceptions.
Interesting to say you've "solve[d] persistence" when you seem to be limited by it here. Is there a particular reason your services can't be architected in less stateful, more 12-factor way?
Kick the persistence can down the road some more? Sure, why not? But sooner or later, somebody has to write something to disk (or flash or whatever that doesn't disappear when the power's off). A system that stores data is inherently stateful. Yes, you can restart services that provide access or auxiliary services (e.g. repair) but the entire purpose of the service as a whole is to retain state. It's the foundation on top of which all the slackers get to be stateless themselves.
The vast majority of people simply redefine the terms to fit whatever they are selling.
If your systems are immutable they can run read-only. In the in nineties Tripwire, the integrity checker, popularized it. You could run it off cdrom. Today immutable infrastructure is VMs/containers that can be ran off a SAN or a pass through file system that is readonly. It means snapshots are completely and immediately replicatable. When you need to deploy, you take a base image/container, install a code onto it, run tests to ensure that it is not broken and replicate it as many times as you need, in a read-only state. This approach also has an interesting property where because system is readonly ( as in exported to the instance read-only/mounted by the instance readonly ) it is extremely difficult to do nasty things to it after a break in - if it is difficult to create files, it is difficult to stage exploits.
That's the only kind of infrastructure where configuration management on the instances themselves is not needed
Yes, but to be clear, some of those containers have been .Net Core containers (running in Kubernetes) for me. I appreciate not having Windows in an estate isn't common to all setups.
> What I prefer to do is use Terraform to create immutable infrastructure from code.
Can you mount all your volumes read-only and run all of your stack? If you cannot, then you do not have immutable infrastructure. You simply happen to agree that no one write anything useful, which with time will absolutely fail because someone, somewhere is going to start storing state on a stateless system giving you "a cow #378 called 'Betsy'"
In the current state of infrastructure, an accepted definition of "immutable infrastructure" is that:
1. You deploy a completely fresh instance/container, instead of in-place updates
2. You don't actively push changes on a running instance/container
Of course you might have stuff written to disk, such as logs, temp files, etc. But it should be non-essential data, and potentially pushed to a central place in near real-time.
I'm going to agree with you. In 2020 (and really the last few years), configuration management is outdated. IaC (infrastructure as code) is the current approach. Containerize everything you can, use terraform or cloudformation, or azure devops.
Avoid managing the underlying os as much as possible. Use vanilla or prebuilt images to deploy these containers on, coreos, Amazon's new bottle rocket (maybe). Or use a service like fargate when possible. All configuration should be declarative to avoid errors.
If you need to build images tools like packer are great. AWS has a recommended "golden Ami pipeline" pattern and a new image builder service if you can't use community images.
I'm speaking imperatively but read these as my own directives. I work for a company that consults and actively helps fortune 500's migrate to the cloud. So some of what I'm saying is not possible or harder on prem and I recognize that.
If I had to, I still like Chef, puppet second favorite mostly because of familiarity. Ansiblee can be used with either of these. And tools like serverspec to validate your images. I don't really use any of this anymore though.
But you still need to configure things, even if they are immutable at runtime. And you need to manage that configuration over time in some systematic way.
You always have a configuration management system.
I'm using Terraform to deploy Docker containers. Terraform's docker_container resource has a lovely 'upload' feature which one can use to upload files into the container. I make Terraform load server config files (or use multi-line strings in the .tf file), perform variable replacement, then destroy Docker containers and recreate them with updated config files. All persistent data is stored in directories bind-mounted into the Docker container.
Terraform has some limitations. For example, one deployment cannot deploy hosts and their containers [1]. And there is no usable support for rolling deployments [2, 3]. So I've ended up with a 4-stage deployment: host-set1, containers on host-set1, host-set2, containers on host-set2.
I also use Terraform to deploy the servers to my laptop during development. Docker for Mac works well.
Someday, Kubernetes will get some usable documentation on how to do normal things [4]. Then I will use it for deploying containers, load balancers, and persistent volumes. For now, it's too big of a complexity jump over plain Docker.
"Amazon/Google/Azure takes care of that" is not the answer, unless your comment is predicated on a world where compute can only be rented from big corps... and their methods of managing underlying infrastructure are sacred secrets for which we are to unworthy to comprehend.
Terraform keeps track of resources it creates. One can remove resources (VMs, managed databases, persistent volumes, DNS records) from the config file and Terraform will cleanly delete them. This is a crucial feature for most deployments.
For example, I deployed an app backend to DigitalOcean with a load balancer, 2x replicated API server, 2x replicated worker process, managed database, file storage, and static website. Terraform is tracking 114 resources for each deployment.
It seems that automatic removal is poorly supported by Ansible, Chef, Puppet, Salt, etc. One can explicitly add items to a "to_remove" list, but this is error-prone.
Terraform has many limitations and problems, but I have found no better tool.
I'll tell you the one tool I DON'T use. Cloudformation. I've touched it a grand total of once and it burned me so hard I set a company policy to never use it again.
It's like terraform, except you can't review things for mistakes until it's already in the process of nuking something. Which is terrible when you're inheriting an environment.
For anyone here who isn't yet using and end to end setup like terraform, ansible, puppet etc and has more basic needs around managing environment variables and application properties, I highly recommend https://configrd.io.
You can never go wrong with bash, you should not put secrets in 169.254.169.254 metadata and you should not have IAM profiles that have overreaching privileges. Any IAM profile that you use or whatever you use on azure or gcp you should always consider what somebody can do with it if they get access to it.
Probably also just straight up docker and docker compose is another good idea, and terraform and possibly hashicorp vault are real high on the list, too. Ansible and chef and puppet are all pretty esoteric and I thought chef was great till I just got good with bash and gnu parallel
Not FAANG but for small to medium "cloud native" businesses I like to use this approach with minimal dependencies:
Managed Kubernetes cluster such as GKE for each environment, setup in cloud provider UI since this is not done often. If you automate it with terraform chances are next time you run it, the cloud provider has subtly changed some options and your automation is out-of-date.
Cluster services repository with Helm charts for ingress controller, centralized logging and monitoring, etc. Use a values-${env}.yaml for environment differences. Deploy with CI service such as Jenkins.
Configuration repository for each application with Helm Chart. If it's an app with one service or all services in a single repo this can go in the same repo. If it's an app with services across multiple repos, create a new repo. Use a values-${env}.yaml for environment differences. Deploy with CI service such as Jenkins.
Store secrets in cloud secrets manager and interpolate to Kubernetes secrets at deploy time.
Cloud provider keeps the cluster and VMs up-to-date, CI pipelines do the builds and deployments. No terraform/ansible/other required. Again, this only works for "cloud native" models.
Yeah, in a decent architecture the only place state is located is in the datastore layer.
The goal is to make servers disposable, able to be destroyed and created at will, so configuration management becomes kind of a legacy technology at that point.
Dev/QA/Similar: Either containerize and back to persistent volume, or use a managed DB service such as RDS or Cloud SQL and create a schema per environment. Include a deployment pipeline argument to reset to known state. CI pipeline can be tuned to handling dynamic environments in either case.
Stage/Prod: Use managed DB service such as RDS or Cloud SQL.
The time and cost to automate a DB upgrade with every edge case considered is huge. Rarely makes sense for small/medium business.
Nitpick: I really don't suggest a divergence in the DB/stack-of-choice between Dev/QA/Stage/Prod. I've chased so many issues that were in the planning process dismissed as "yeah that's an edge case and most likely won't happen".
The reasons I've seen for doing so are usually penny-wise, pound-foolish. Penny-wise in saving a few dollars (conceptually) on a spreadsheet for per-env/per-cycle, while neglecting the long-tail consequence of your labor factor just growing, potentially forever, without regard for total cost of ownership.
Our setup is quite similar to this. Some differences - each environment is represented as a helm parent chart with each application being a child chart. Each environment chart has it's own repo where values.yaml supplies environment specific overrides for each application. Each application has its own repo where the helm charts and application source both reside.
Applications have a unique source repo. Each rep has a build dir. Build dir contains a sundir for docker build, terraform configs (if needed) for dependent infrastructure and a helm chart for deploy.
I have a few things that don't fit the microservice pattern. They are terraform first (root of repo is TF code) and they have a build dir to define the next steps (mostly packer).
As I'm writing this, I think I need to change that and make the top level a README.md file and use the build dir pattern to be consistent.
they had a very large presence at RSAC 2020. i was very confused by it. they are not security software. however i suppose there are no “configuration management “ tradeshows
I haven't looked into what they added specifically for the space, but I think a configuration management company at a security trade show makes sense: Configuration management is a very useful tool for various security goals.
> Salt is a new approach to infrastructure management built on a dynamic communication bus. Salt can be used for data-driven orchestration, remote execution for any infrastructure, configuration management for any app stack, and much more.
I'm ultra confused about their marketing btw. Their website doesn't even say it's open source. You have to sign up to "try it now". It's like they don't want customers? Or are people who want to understand what they're buying not the target market, somehow?
Sure, but understanding what the thing is, that's part of the buying decision right? I have no clue what https://www.saltstack.com/ is about.
How do I get from this:
> Drive IT security into the 21st century. Amplify the impact of your entire SecOps team with global orchestration and automation that remediates security issues in minutes, not weeks.
to "it's a provisioning tool for servers, like ansible but faster"?
The docs landing page have me an easier to grok picture of what it does, purely based on seeing what the doc headers are - https://docs.saltstack.com/en/latest/
Salt has much nicer configs and feels supeior to Ansible. The main disadvantage that I had with salt was the need to have a salt master server. I read that this is no longer needed but I have not tried it myself. Keeping secrets outside of the repo was not trivial task, ansible has an easy way to encrypt secrets.
Made worse by the same thing that plagues Homebrew: making up cutesy vocabulary for things. I've been working with computers for a long time, Salt, you can feel free to use Adult Words when describing a thing to me
Ansible for dev boxes or smaller deployments.
For large-scale deployments CFEngine3. When deployed within a cloud environment one doesn't even need a master node for CFE3 but the agents can just pull the latest config state from some object storage.
Hashicorp tools are quite solid, and give you a lot for free. Ansible can automate host-level changes in places where hashicorp cannot reach. There shouldn't be many such places.
Alternatively, if you have the option of choosing the whole stack, Nix/NixOS and their deployment tools.
I would recommend staying away from large systems like k8s.
It is silly to ask "what should be used at FAANG scale", because either you are working at a FAANG and you are using what they use, or you are very unlikely to ever be at that scale -- and somewhere along the journey to getting there, you will either find or write the system that you need.
Odd reply to my statement that FAANGs roll their own config management tools. Are you smoking a lot of weed during this quarantine? You can google linkedins version of docker 'locker', facebooks hydra, etc etc. Alot of these big companies roll their own tools because consumer tools aren't scale enough. Linked In rolled Kafka for just this reason as well.
Also, odd and a little creepy that you're stalking my comments but I hope you enjoyed my brain droppings. ;)
Here also docker-compose. Easy to separate tenants using same stack (nginx+django+postgres+minio).
Question though: how do you manage the possible rebooting-containers-loop after a host reboot? I had to throw in more memory to prevent this but it feels like a (expensive|unnecessary) workaround. Anyone figured out how to let multiple containers start after each other (while not in 1 docker-compose.yaml)?
I've prototyped ansible for rolling out ssl certs to a handful of unfortunately rather heterogeneous Linux boxes - and it worked pretty well for that.
I still think there's too much setup to get started - but am somewhat convinced ansible does a better job than a bunch of bespoke shell would (partly because ansible comes with some "primitives"/concepts such as "make sure this version of this file is in this location on that server - which is quick to get wrong across heterogeneous distributions).
We're moving towards managed kubernetes (for applications currently largely deployed with Docker and docker-compose on individual vms).
I do think the "make an appliance;run an appliance;replace the appliance" life cycle makes a lot of sense - I'm not sure if k8s does yet.
I think we could be quite happy on a docker swarm style setup - but apparently everything but k8s is being killed or at least left for dead by various upstream.
And k8s might be expensive to run in the cloud (a vm pr pod?) - but it comes with abstractions we (everyone) needs.
Trying to offload to SaaS that which makes sense as SaaS - primarily managed db (we're trying out elephant sql) - and some file storage (100s of MB large Pdf files).
For bespoke servers we lean a bit on etckeeper in order to at least keep track of changes. If we were to invest in something beyond k8s (it's such a big hammer, that one become a bit reluctant to put it down once picked up..) I'd probably look at gnu guix.
I'm pretty happy using both Puppet and ansible. I use Puppet for configuring hosts and rolling out configuration changes (because immutable infrastructure isn't a thing you can just do; there's overhead and it does not fit all problems) and ansible for orchestrating actions such as upgrades. They work well together.
I very much dislike ansible's YAML-based language and would hate to use it for configuration management beyond tiny systems, but it's pretty decent as a replacement for clusterssh and custom scripts.
I'm using puppet for everything, including nearly immutable infrastructure ( if you can't mount your disks read only and run that way you dont have immutable infrastructure )
Puppet maintains the base image with the core system.
Special systems are recreated by applying system specific classes to a base image.
Application software is installed via packages with git commit-ids being versions.
Nothing is upgraded, rather a new instances are rolled out and the old instances are destroyed.
This also ensures that we always know that we can recreate our entire infrastructure because we do that for rapidly changing systems several times a day and for all systems at least monthly.
This makes our operational workflow match the disaster recovery, which is god sent.
217 comments
[ 3.6 ms ] story [ 267 ms ] threadhttps://cuelang.org/
(To me things like puppet or ansible seem like thin layers over shell and ssh, whereas both Dhall and Cue seem to innovate in ways that are more, uh, je ne sais quoi ;-) YMMV)
Or Dockerfile/compose for container images.
Cloud resources are managed by Terraform/Terragrunt.
My experience trying to manage longer lived systems like robot computers over time with Ansible has been that it quickly becomes a nightmare as your playbook grows cruft to try to account for the various states the target may be coming from.
I still use it as it allows more portability across OS releases and families (as in easier migration), but it also increases the complexity when creating a new task/role/playbook.
In that sense, Dockerfiles with shell-based RUN commands are much easier to manage.
Another advantage of config management over shell might be a better integration with the underlying cloud provider. For instance Ansible supports AWS SSM parameter store, which allows me to use dynamic definitions of some configuration data (RDS database endpoints, for instance) or secrets (no need for Ansible vault)
Even in the run-once case, though, I think there's a benefit to Ansible's role-based approach to modularization. And again for the dev scenario, it's much easier to run only portions of a playbook than it is to run portions of a shell script.
And finally, the diagnostics and overall failure story are obviously way better for Ansible, too.
Now, all this said, I do still go back and forth. For example, literally right now in another window I'm working a small wrapper that prepares clean environments to build patched Ubuntu kernels in— and it's all just debootstrap, systemd-nspawn, and a bunch of shell script glue.
https://www.ansible.com/ is surely a good solution for Bootstraping Linux cloud machines and can be quite flexible. I personally feel like its usage of YAML manifests instead of a domain-specific language can make complex playbooks harder to read and to maintain.
If all you do is to deploy containers on a managed Kubernetes or a similar platform, you might get away with some solution to YAML templating (jsonnet et al) and some shell glue.
I am keeping an eye on https://github.com/purpleidea/mgmt which is a newer contender which many interesting features but lacks more complex examples.
Others like saltstack and chef still see some usage as far as I know, but I've got no personal experience with them.
It's still good in Enterprise land where taking the time to work out the declarative style and dependency chains is worth it (and you have the people to put on it and the CAB process to review infrastructure changes). For a small to mid sized company I find it gets in the way of iterating fast. I spent waaaay too much time there either fighting the tooling or having to work out dependency chains. Redhat and I-think-AWS-but-I-might-be-thinking-of-Chef also have tooling in this space.
I'll take Chef or Ansible's imperative approach in the environments I work in any day (Mostly ansible playbooks for baking hosts only, I've never been entirely comfortable with having one Ansible Tower/Chef Server/Puppetmaster/etc be authoritative over everything, too large a failure pattern if security controls fail). But again, I'm working in many younger small environments and not large mature ones.
Most of this is also irrelevant for us as we're all in on Docker/ECS for anything new. Config management plays a limited role there over having your tasks/services checked into the individual repos.
As you mention, puppet has a steep learning curve, whereas Ansible has a very shallow one. It’s easy to get running in a few minutes!
We use both Puppet and Ansible at work, and its constant complaints and delays with Puppet whereas Ansible is little complaints and no delays.
That's probably because you are not running masterless, which means your puppet master is a bottle neck.
I would argue that masterless puppet is a superior pattern for both scaling creating a hierarchical structures.
So it's a very good tool to gradually get a legacy system under configuration management and thus source control.
What I prefer to do is use Terraform to create immutable infrastructure from code. CoreOS and most Linux variants can be configured at boot time (cloud-config, Ignition, etc) to start and run a certain workload. Ideally, all of your workloads would be containerised, so there's no need for configuration drift, or for any management software to be running on the box. If you need to update something, create the next version of your immutable machine and replace the existing ones.
Are Ansible/Puppet/Chef any good for managing the hardware you mentioned?
Yes. Well, OK, maybe not good but better than ad hoc.
Ansible is used in networking, which many vendors having official modules:
* https://github.com/PaloAltoNetworks/ansible-pan
* https://github.com/aristanetworks/ansible-cvp
* https://www.juniper.net/documentation/en_US/junos-ansible/in...
* https://github.com/F5Networks/f5-ansible
There are even frameworks if you want to write things in 'raw' Python as well.
(although that's of course not strictly "old stuff")
Rather than doing that I take a light touch approach with ansible that will suffice until we can dockerize (which would require the same work, but then it’s a dev project vs now when it’s just a devops thing)
(Context: my team manages dozens of clusters, each with a score of services across thousands of physical hosts. Every minute of every day, multiple things are being scaled up or down, tuned, rearranged to deal with hardware faults or upgrades, new features rolled out, etc. Far from being immutable, this infrastructure is remarkably fluid because that's the only way to run things at such scale.)
Beware of Chesterton's Fence. Just because you haven't learned the reasons for something doesn't mean it's wrong, and the new shiny often re-introduces problems that were already solved (along with some of its own) because of that attitude.
As for the conflicts, I have to say I loathe the way the more dynamic part of configuration works. It might be the most ill conceived and poorly implemented system I've seen in 30+ years of working in the industry. Granted, it does basically work, but at the cost of wasting thousands of engineers' time every day. The conflicts occur because (a) it abuses source control as its underlying mechanism and (b) it generates the actual configs (what gets shipped to the affected machines) from the user-provided versions in a non-deterministic way which causes spurious differences. All of its goals - auditability, validation, canaries, caching, etc. - could be achieved without such aggravation if the initial design hadn't been so mind-bogglingly stupid.
But I digress. Sorry not sorry. ;) To answer your question, my personal solution is to take advantage of the fact that I'm on the US east coast and commit most of my changes before everybody else gets active.
State is always the enemy in technology.
I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
I work in data storage. Am I the enemy, then? ;)
> can't even imagine managing hundreds of servers whose state is unpredictable at any moment
Be careful not to conflate immutability with predictability. The state of these servers is predictable. All of the information necessary to reconstruct them is on a single continuous timeline in source control. But that doesn't mean they're immutable because the head of that timeline is moving very rapidly.
> can't be terminated and replaced with a fresh instance for fear of losing something.
No, there's (almost) no danger of losing any data because everything's erasure-coded at a level of redundancy that most people find surprising until they learn the reasons (e.g. large-scale electrical outages). But there's definitely a danger of losing availability. You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming. Rolling changes are an absolute requirement. Some take minutes. Some take hours. Some take days. Many of these services have run continuously for years, barely resembling the code or config they had when they first started, and their users wouldn't have it any other way. It might be hard to imagine, but it's an every-day reality for my team.
You’re the prison guard.
I don't trust predictability. Drift is always a nightmare. Nothing is ever as predictable as you would like it to be.
>You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming.
If it's architected well you can :)
Except that state and its manipulation is usually the primary value in technology.
> I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
Yes, that sounds awful. That's why we have backups and, if necessary, redundancy and high availability.
Exactly and thats why you put state in data stores and keep your servers immutable.
My understanding of immutable infrastructure is the same as immutable data structures: once you create something, you don't mess with it. If you need a different something, you create a new one and destroy the old one.
That doesn't mean that the whole picture isn't changing all the time. Indeed, I think immutability makes systems overall more fluid, because it's easier to reason about changes. Mutability adds a lot of complexity, and when mutable things interact, the number of corner cases grows very quickly. In those circumstances, people can easily learn to fear change, which drastically reduces fluidity.
It seemed like a fine idea to me. The best way to be sure that everything can be rebuilt is to regularly rebuild everything. It also solves some security problems, simplifies maintenance, and allows people to be braver around updates.
And just to be clear, I'm very willing to believe that your particular legacy setup isn't a good match for cattle-not-pets practices. But I think that's different than saying it's impossible for anybody to bring an immutable approach to things like storage.
To give a really silly example, adding a node to a cluster is a configuration change. It wouldn't make sense to destroy the cluster and recreate it to add a new node. There are lots of examples like this where if you took the idea of immutable infrastructure to the extreme it would result in really large wastes of effort.
On the other hand, while my suggestion of doing it to add a node sounds ridiculous I'm sure there are circumstances in which it's not only understandable but necessary, due to some aspect of the system.
If your biggest objection to an idea is that you can make up a silly thing that sounds like it might be related, I'm not understanding why we need to have this discussion. I'd like to focus on real issues, thanks.
I also think it's a cool idea to destroy the entire cluster just to add a node, and it sounds ridiculous but also like there's some circumstances where it makes perfect sense.
If your systems are immutable they can run read-only. In the in nineties Tripwire, the integrity checker, popularized it. You could run it off cdrom. Today immutable infrastructure is VMs/containers that can be ran off a SAN or a pass through file system that is readonly. It means snapshots are completely and immediately replicatable. When you need to deploy, you take a base image/container, install a code onto it, run tests to ensure that it is not broken and replicate it as many times as you need, in a read-only state. This approach also has an interesting property where because system is readonly ( as in exported to the instance read-only/mounted by the instance readonly ) it is extremely difficult to do nasty things to it after a break in - if it is difficult to create files, it is difficult to stage exploits.
That's the only kind of infrastructure where configuration management on the instances themselves is not needed
Can you mount all your volumes read-only and run all of your stack? If you cannot, then you do not have immutable infrastructure. You simply happen to agree that no one write anything useful, which with time will absolutely fail because someone, somewhere is going to start storing state on a stateless system giving you "a cow #378 called 'Betsy'"
1. You deploy a completely fresh instance/container, instead of in-place updates 2. You don't actively push changes on a running instance/container
Of course you might have stuff written to disk, such as logs, temp files, etc. But it should be non-essential data, and potentially pushed to a central place in near real-time.
There's no point in trying to manage a database or similar resources this way.
Avoid managing the underlying os as much as possible. Use vanilla or prebuilt images to deploy these containers on, coreos, Amazon's new bottle rocket (maybe). Or use a service like fargate when possible. All configuration should be declarative to avoid errors.
If you need to build images tools like packer are great. AWS has a recommended "golden Ami pipeline" pattern and a new image builder service if you can't use community images.
I'm speaking imperatively but read these as my own directives. I work for a company that consults and actively helps fortune 500's migrate to the cloud. So some of what I'm saying is not possible or harder on prem and I recognize that.
If I had to, I still like Chef, puppet second favorite mostly because of familiarity. Ansiblee can be used with either of these. And tools like serverspec to validate your images. I don't really use any of this anymore though.
You always have a configuration management system.
Terraform has some limitations. For example, one deployment cannot deploy hosts and their containers [1]. And there is no usable support for rolling deployments [2, 3]. So I've ended up with a 4-stage deployment: host-set1, containers on host-set1, host-set2, containers on host-set2.
I also use Terraform to deploy the servers to my laptop during development. Docker for Mac works well.
Someday, Kubernetes will get some usable documentation on how to do normal things [4]. Then I will use it for deploying containers, load balancers, and persistent volumes. For now, it's too big of a complexity jump over plain Docker.
[1] https://github.com/hashicorp/terraform/issues/2430
[2] https://github.com/hashicorp/terraform/issues/23735#issuecom...
[3] https://github.com/hashicorp/terraform/issues?q=is%3Aissue+%...
[4] https://github.com/kubernetes/website/issues/19139
"Amazon/Google/Azure takes care of that" is not the answer, unless your comment is predicated on a world where compute can only be rented from big corps... and their methods of managing underlying infrastructure are sacred secrets for which we are to unworthy to comprehend.
For example, I deployed an app backend to DigitalOcean with a load balancer, 2x replicated API server, 2x replicated worker process, managed database, file storage, and static website. Terraform is tracking 114 resources for each deployment.
It seems that automatic removal is poorly supported by Ansible, Chef, Puppet, Salt, etc. One can explicitly add items to a "to_remove" list, but this is error-prone.
Terraform has many limitations and problems, but I have found no better tool.
[0] https://www.chef.io/customers/facebook/ [1] https://engineering.fb.com/data-center-engineering/tupperwar...
It's like terraform, except you can't review things for mistakes until it's already in the process of nuking something. Which is terrible when you're inheriting an environment.
https://github.com/DBCDK/morph
Managed Kubernetes cluster such as GKE for each environment, setup in cloud provider UI since this is not done often. If you automate it with terraform chances are next time you run it, the cloud provider has subtly changed some options and your automation is out-of-date.
Cluster services repository with Helm charts for ingress controller, centralized logging and monitoring, etc. Use a values-${env}.yaml for environment differences. Deploy with CI service such as Jenkins.
Configuration repository for each application with Helm Chart. If it's an app with one service or all services in a single repo this can go in the same repo. If it's an app with services across multiple repos, create a new repo. Use a values-${env}.yaml for environment differences. Deploy with CI service such as Jenkins.
Store secrets in cloud secrets manager and interpolate to Kubernetes secrets at deploy time.
Cloud provider keeps the cluster and VMs up-to-date, CI pipelines do the builds and deployments. No terraform/ansible/other required. Again, this only works for "cloud native" models.
The goal is to make servers disposable, able to be destroyed and created at will, so configuration management becomes kind of a legacy technology at that point.
Dev/QA/Similar: Either containerize and back to persistent volume, or use a managed DB service such as RDS or Cloud SQL and create a schema per environment. Include a deployment pipeline argument to reset to known state. CI pipeline can be tuned to handling dynamic environments in either case.
Stage/Prod: Use managed DB service such as RDS or Cloud SQL.
The time and cost to automate a DB upgrade with every edge case considered is huge. Rarely makes sense for small/medium business.
The reasons I've seen for doing so are usually penny-wise, pound-foolish. Penny-wise in saving a few dollars (conceptually) on a spreadsheet for per-env/per-cycle, while neglecting the long-tail consequence of your labor factor just growing, potentially forever, without regard for total cost of ownership.
Sorry didn't mean to rant. Hope this helps.
[1] https://github.com/boxboat/dockhand
Applications have a unique source repo. Each rep has a build dir. Build dir contains a sundir for docker build, terraform configs (if needed) for dependent infrastructure and a helm chart for deploy.
I have a few things that don't fit the microservice pattern. They are terraform first (root of repo is TF code) and they have a build dir to define the next steps (mostly packer).
As I'm writing this, I think I need to change that and make the top level a README.md file and use the build dir pattern to be consistent.
When working solo I use Guix, both Guix and Nix are _seriously_ amazing.
Salt (also known as SaltStack) was right.
> Salt is a new approach to infrastructure management built on a dynamic communication bus. Salt can be used for data-driven orchestration, remote execution for any infrastructure, configuration management for any app stack, and much more.
I'm ultra confused about their marketing btw. Their website doesn't even say it's open source. You have to sign up to "try it now". It's like they don't want customers? Or are people who want to understand what they're buying not the target market, somehow?
For reference, this appears to be the Salt primer: https://docs.saltstack.com/en/getstarted/system/index.html
How do I get from this:
> Drive IT security into the 21st century. Amplify the impact of your entire SecOps team with global orchestration and automation that remediates security issues in minutes, not weeks.
to "it's a provisioning tool for servers, like ansible but faster"?
https://alcali.dev
feedback welcome!
Alternatively, if you have the option of choosing the whole stack, Nix/NixOS and their deployment tools.
I would recommend staying away from large systems like k8s.
It is silly to ask "what should be used at FAANG scale", because either you are working at a FAANG and you are using what they use, or you are very unlikely to ever be at that scale -- and somewhere along the journey to getting there, you will either find or write the system that you need.
;)
Also, odd and a little creepy that you're stalking my comments but I hope you enjoyed my brain droppings. ;)
Question though: how do you manage the possible rebooting-containers-loop after a host reboot? I had to throw in more memory to prevent this but it feels like a (expensive|unnecessary) workaround. Anyone figured out how to let multiple containers start after each other (while not in 1 docker-compose.yaml)?
I still think there's too much setup to get started - but am somewhat convinced ansible does a better job than a bunch of bespoke shell would (partly because ansible comes with some "primitives"/concepts such as "make sure this version of this file is in this location on that server - which is quick to get wrong across heterogeneous distributions).
We're moving towards managed kubernetes (for applications currently largely deployed with Docker and docker-compose on individual vms).
I do think the "make an appliance;run an appliance;replace the appliance" life cycle makes a lot of sense - I'm not sure if k8s does yet.
I think we could be quite happy on a docker swarm style setup - but apparently everything but k8s is being killed or at least left for dead by various upstream.
And k8s might be expensive to run in the cloud (a vm pr pod?) - but it comes with abstractions we (everyone) needs.
Trying to offload to SaaS that which makes sense as SaaS - primarily managed db (we're trying out elephant sql) - and some file storage (100s of MB large Pdf files).
For bespoke servers we lean a bit on etckeeper in order to at least keep track of changes. If we were to invest in something beyond k8s (it's such a big hammer, that one become a bit reluctant to put it down once picked up..) I'd probably look at gnu guix.
I very much dislike ansible's YAML-based language and would hate to use it for configuration management beyond tiny systems, but it's pretty decent as a replacement for clusterssh and custom scripts.
Puppet maintains the base image with the core system.
Special systems are recreated by applying system specific classes to a base image.
Application software is installed via packages with git commit-ids being versions.
Nothing is upgraded, rather a new instances are rolled out and the old instances are destroyed.
This also ensures that we always know that we can recreate our entire infrastructure because we do that for rapidly changing systems several times a day and for all systems at least monthly.
This makes our operational workflow match the disaster recovery, which is god sent.